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ABSTRACT 



This resource guide, the first of three volumes, lays out 
the fundamentals of outcome measurements for assistive technology. It 
includes the whys and hows of gathering data so that assistive technology 
practitioners can integrate outcomes measurement activities in their daily 
practice. Chapters include: (1) "Concepts and Rationale for Accountability in 

Assistive Technology" (Frank DeRuyter) , which addresses scope of assistive 
technology practice and need for quality and assistive technology 
stakeholders of quality; (2) "Accountability in Assistive Technology" (Roger 
0. Smith) , which discusses the hierarchy in the complexity of assistive 
technology outcomes, major domains of outcome, types of performance, the 
taxonomy of outcomes, and outcome measures; (3) "Program Evaluation" (Jan 
Galvin) ; (4) "Characteristics of a Meaningful Outcome Assessment" (Marcia J. 

Scherer) ; and (5) "Where Are We Headed with Assistive Technology Outcomes?" 
(Roger O. Smith and Marcia J. Scherer), which addresses issues in the 
measurement of assistive technology outcomes and selecting the outcomes to 
measure. An annotated bibliography is provided on assistive technology and 
outcomes assessment that contains 40 resources. (Each chapter includes 
references.) (CR) 
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VOLUME I - RESNA RESOURCE GUIDE FOR 
ASSISTIVE TECHNOLOGY OUTCOMES: 
MEASUREMENT TOOLS 



INTRODUCTION 



This document is the first of the three volume set of the RESNA Guide for Assistive 
Technology Outcomes. The volumes are as follows: 

VOLUME I: RESNA RESOURCE GUIDE FOR ASSISTIVE TECHNOLOGY 

OUTCOMES: MEASUREMENT TOOLS 

This is an easy-to-read text that lays out the fundamentals of outcomes measurement. It includes 
the whys and hows of gathering data so an A.T. practitioner can integrate outcomes measurement 
activities in their daily practice. 



VOLUME II* RESNA RESOURCE GUIDE FOR ASSISTIVE TECHNOLOGY 

OUTCOMES: ASSESSMENT INSTRUMENTS, TOOLS, & 
CHECKLISTS FROM THE FIELD 

Volume II is a compilation of assessment instruments, tools, or checklists. These instruments 
were submitted by active professionals in the field of assistive technology and demonstrate the 
range of instrumentation in use today. Each item submitted is reviewed according to a format. It 
is anticipated that this will become an evolving document. 



VOLUME III: RESNA RESOURCE GUIDE FOR ASSISTIVE TECHNOLOGY 

OUTCOMES: DEVELOPING DOMAINS OF NEED AND 
CRITERIA OF SERVICES 

Volume III provides explanation of domains of anticipated assistive technology impact across 
functional areas of an individual's life. 
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CONCEPTS AND RATIONALE FOR ACCOUNTABILITY 
IN ASSISTIVE TECHNOLOGY 

Frank DeRuyter, Ph.D. 

"Each day, each product or service is getting relatively better 
or relatively worse, but it never stands still. " 

Tom Peters 

The relationship between the delivery of goods and/or services and the consequential 
outcomes has been long recognized. Throughout the past century, there have been many examples 
in which the manufacturing and service delivery sectors have strongly advocated the employment of 
various objective measurements to improve upon the delivery of goods and services. In recent 
years, the escalating importance that society has ordered on accountability, performance monitoring 
and personal responsibility has led to an increased emphasis being placed on the monitoring of 
service delivery practices and the evaluation of outcomes. This renewed emphasis however, has 
placed certain areas of the service delivery sector into a state of confusion because it is usually 
harder to evaluate the services than the goods. One cannot always "kick the tires before a service 
is delivered. By the same token, it is equally difficult to "return a service" to a complaint 
department. Within the assistive technology arena, the various stakeholders have begun to realize 
the validity of this statement as well as the accompanying difficulties in the development of 
performance monitoring and accountability systems as it responds to this dilemma. 



OVERVIEW 

The demand for assistive technology services continues to grow at a very rapid pace. This 
is in light of the fact that the costs for assistive technology have become significant and are 
frequently a lifelong expense. During the past three years, the clinical service delivery sector 
throughout the United States has witnessed several significant events that raise serious questions 
about the availability of resources in the future. These events have included: (1) a political 

realignment to a new national public policy agenda shifting from a social-ethical agenda where 
consumers have had access to goods and services irrespective of cost to an economic agenda where 
the access to goods and services is being driven by cost and a pricing paradigm; (2) an ongoing 
national expenditure crisis; and, (3) dramatic changes to the global economy. The potential 
consequence of each of these events is that for the first time the assistive technology community 
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stands to find itself at great risk about whether it will be able to deliver its current level of goods 
and services in the future. Especially in light of the fact that the dollars available in the future are 
most likely to decline. One can only hope for the best case scenario which will be that the current 

dollars available to meet the demand will remain unchanged. 

For the reasons stated above, it becomes imperative that the various stakeholders within the 
assistive technology community identify alternative and more efficient ways of delivering goods 
and services as well as begin developing performance monitoring systems in order to be held to a 
higher degree of accountability than has been executed to date. The providers of assistive 
technology goods and services must demonstrate positive outcomes of high value and at a 
reasonable price. This will necessitate that all assistive technology stakeholders come to terms with 
whether the expenses associated with assistive technology goods and services justify the benefits 
gained. Accountability today will minimize the denial of goods and services tomorrow. 

Accountability and performance monitoring within the assistive technology clinical service 
delivery sector will provide the qualitative and quantitative data that evaluates the 
comprehensiveness, effectiveness and the efficacy of the services provided. This information 
should specifically ascertain the value as well as best practices of practitioners, services, products 
and programs alike. At a minimum, the evaluation process must measure and establish a baseline 
of what works; identify how well and for whom it works; and, determine at what level of economic 
efficiency it works. The process whereby data are selected to answer these questions will require 
taking information from several performance monitoring dimensions. This process has also been 
referred to as outcomes management. In the clinical service delivery sector, frequently used 
dimensions have included clinical status/results; functional status; quality of life; satisfaction; and, 

cost. 

Accountability, performance monitoring and outcomes management is frequently conducted 
within the framework of continuous quality improvement (CQI). Data are correlated with specific 
processes or systems that may be modified, thereby enhancing the effectiveness and efficiency of 
service delivery, ultimately improving the outcome. This process has also more recently been 
referred to as total quality management (TQM). Measuring performance and outcomes provides 
the greatest indication that process or system changes are in fact improving quality levels. Over the 
past several years, numerous clinical service delivery environments have begun to clearly 
demonstrate that these initiatives have actually improved patient/client outcomes and have 
decreased the variations in clinical practice. Despite the conventional wisdom that better quality 
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costs more, it has been determined that in the long run, good care usually costs less than poor care. 

In fact, several leading practitioners claim that the best way to reduce costs is to improve quality 
(DeRuyter, 1997). 

RATIONALE FOR ACCOUNTABILITY 

Quality mandate in assistive technology 

The q ual ity mandate and its focus are not a recent phenomenon. It has been said on 
numerous occasions that quality is an entity that exists in the eye of the beholder and that therefore 
quality is simply a perception. With that said, however, it is also important to acknowledge that 
perceptions form expectations that become the realities. The assistive technology community must 
therefore measure as well as manage those perceptions in order to be accountable. Furthermore, 
this accountability must be accepted in all aspects of assistive technology and by all stakeholders. 

Over the past century, many have advocated different objective measures to improve upon 
the delivery of various products and services within the manufacturing sector. In the clinical 
service delivery arena, Florence Nightingale is probably most widely credited with incorporating 
the first scientific approach at evaluating quality. It has only been over the past three decades, 
however, that an increased importance has been placed on accountability and performance 
monitoring as it relates to clinical activities within the healthcare and educational sectors. This 
monitoring has focused primarily on trying to measure the quality of care provided, although 
recently the emphasis has shifted toward measuring clinical and cost outcomes. Ironically, the 
assistive technology community has generally exhibited reluctance at being held accountable until 
just recently. This has been in spite of several assistive technology stakeholders that come from 
sectors that have embraced performance monitoring for a number of years. 

Much of the present day quality mandate within the clinical service delivery sector has been 
driven by activities that began in the late 1960's. These early efforts were in fact retrospective 
utilization plans adopted by both healthcare and education. These early quality measurement 
activities led to the development of professional standards review organizations in the 1970's which 
attempted to evaluate the quality of service provided. These efforts rapidly led to the development 
of peer review organizations and eventually quality assurance programming. Both of these efforts 
were designed to monitor the quality of services provided, however, these efforts only focused on 
chart audits and client satisfaction surveys designed to ascertain what people were complaining 
about and/or what had gone wrong. Great concern was articulated by clinical service providers 
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during this period because they believed they were being asked to participate in a process that in 
reality could never be assured. Despite the quality decrees of the 70's and 80's, clinical service 
providers had few compelling reasons to be accountable or to measure and demonstrate quality. 
Consequently, performance monitoring was frequently ignored, as there were few incentives to 
account for "doing the right thing" let alone examine the effectiveness of the clinical services that 
one delivered. As a result, quality was acclaimed as an idealistic goal that was too difficult to 
measure, let alone define, and accountability remained as an esoteric concept. 

By the mid 1980's, the attitude toward accountability and quality began to change. Many 
service delivery sectors witnessed economic difficulties associated with the national deficit 
spending crisis. This resulted in an era of diminished resources within the education and healthcare 
sectors. As policy changes and budget cuts impacted upon these sectors, it was frequently argued 
that quality would be severely impacted. However, the laissez-faire attitudes that were asserted 
toward quality during the 1970's and 1980's meant that there was only minimal data, often 
anecdotal, to support this contention. Consequently, the incentives to survive overshadowed doing 
the right thing. The end result to the clinical service delivery sector was devastating for both 
consumers and providers. This was evidenced through widespread organizational reengineering; 
denied or limited access to services; poorer quality; and even the outright elimination of services. 

By the late 1980's, consumers were quick to realize the impact of what was occurring which 
resulted in a major public outcry. This marks another period of major change. For clinical service 
providers the impact was referred to as a paradigm shift and focused specifically on numerous new 
requirements. These led to the monitoring of program activities as well as for the first time the 
monitoring and accountability of providers. These changes ultimately became known as quality 
assessment and improvement. More recently, this has been referred to as performance monitoring. 
Today, care provided by the clinical service delivery sector is being managed and evaluated on an 
ongoing basis in order to minimize the associated costs and to maximize the desired outcomes. 
Furthermore, clinical service provider agencies and facilities must now include providers and 
consumers as active participants throughout the entire ongoing evaluation process. 

Scope of assistive technology practice and need for quality 

It has been previously stated that the assistive technology community was able to remain on 
the fringe of the performance monitoring and accountability agenda for several years. It has been 
debated whether this has been an asset or a liability to the assistive technology community. 
Irrespective of the merits of the arguments, it is important to acknowledge that there are some very 
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specific aspects of the scope of assistive technology service delivery practice that have had a 
profound influence over the ability to effectively participate in the performance monitoring and 
accountability agenda let alone agree on the need for quality. By not having adequately developed 
performance monitoring systems, serious questions regarding the effectiveness and efficiency of the 
assistive technology service delivery system as well as the efficacy of the provision of these 
services have been raised. There are those within the assistive technology community who have 
simply operated under the assumption that accountability and improved outcomes are derived 
through better technological solutions. As a result, technological solutions have been looked 
toward for improved outcomes without data to support the assumption. With the knowledge that 
almost one third of all devices are likely to be abandoned (Phillips, 1992), this view has been rather 
myopic. With the excitement toward high tech solutions moderating, a renewed emphasis on 
performance monitoring that examines the validity and benefits of the technological solutions 
appears to be taking place. 

Over the past few years, numerous conference presentations and several publications have 
begun to address performance monitoring as it relates to assistive technology. Although there is 
still a paucity of information, the assistive technology community has not entirely been without 
performance monitoring activities. At a system level, several articles have addressed the need to 
establish some type of the accountability (Blackstone, 1990; DeRuyter, 1992a, 1992b, 1995; Jutai et 
al, 1996; NIDRR, 1994; Smith, 1992; Warren, 1992, 1993). These efforts have been focused at 
several areas including those that were service specific (AAC, access, mobility, etc.); institutional 
specific (pediatric rehabilitation); organizational specific (RESNA); and, agency specific (NIDRR). 
All of these efforts, however, began after there was a legislated mandate to provide services and 
equipment. Consequently, the assistive technology service delivery community has been "in the 
position of having to deal with the cart being before the horse" (Warren, 1993). 

There have also been a number of single case studies or small sample size reports. Some of 
the major studies include the following: a follow-up survey of prototype vocational aids for the 
blind (Brabyn, 1981); a follow-up survey of the various devices still being used 16 weeks after 
device delivery (Caudrey & Seeger, 1983); a retrospective study of client satisfaction with device 
function and comfort five years after delivery (Kohn et al, 1983); a telephone survey of device 
usage two weeks after delivery (McGrath et al, 1985); device utilization and quality of life (Scherer, 
1988); a follow-up on the utilization of devices in rural areas (Willkomm, 1988); what separates 
high tech users from non-users (Scherer & McKee, 1990); a national survey on technology 
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abandonment (Phillips & Zhao, 1993); and post discharge device usage in relation to functional 
status (Cushman & Scherer, 1996). The greatest shortcoming of most of these reports is that they 
have consisted of retrospective reviews rather than prospective studies. As a result, these studies, 
which have been informative, have provided little insight into how to change the manner by which 
services are delivered or how to improve upon the quality of services provided. 

In terms of long-term prospective assistive technology related performance monitoring only 
a handful of studies have been reported. These have included the following: the application and use 
of augmentative and alternative communication systems for 50 individuals who were nonspeaking 
as the result of traumatic brain injury and stroke (DeRuyter & Kennedy, 1990; DeRuyter, Kennedy 
& Doyle, 1990). Both of these studies impacted the delivery of services to these populations as 
they altered the traditional manner of providing AAC systems to all who were nonspeaking in these 
populations by identifying key variables as to when systems should be considered. In another 
prospective study, user feedback as it related to assistive technology was documented through 
standardized data collection at 5 different service delivery centers. These data were used to 
demonstrate that user satisfaction and device performance data could be collected and compared 
across service delivery sites with reliability (Kohn, LeBlanc & Mortola, 1994). In a more recent 
study, post discharge device use of 47 individuals in relation to their functional status was 
addressed (Cushman & Scherer, 1996). This report went on to document the discrepancies in 
perception between therapists and consumers regarding utility and aesthetic aspects of devices as 
well- as provide strategies to maximize appropriate device use. Finally, a recently completed 
longitudinal outcome study, followed over 1 00 children and adolescents each for over a two year 
period to examine the issues of usage and appropriateness of assistive technology as well as the 
underlying and contributing elements that ultimately influenced the outcomes (DeRuyter, in press). 
Each of these studies, which focused on some aspect of performance monitoring, are important to 
the assistive technology community because they have influenced and changed the way in that 
services are provided. It is these long-term, prospective type assistive technology-related 
performance monitoring studies that will effect positive changes in the delivery of assistive 
technology clinical services. 

The assistive technology community has actually witnessed several other subtle efforts that 
have addressed the quality mandate. Probably most familiar is the "State Tech Act" legislation. In 
the 1994 amended reauthorization (PL 103-218) of the enabling legislation, Sections 102(e)(6-8) 
specify that recipients of the discretionary "Tech Act" grants must conduct annual assessments of 
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their program initiatives. In another effort, the American Medical Association in 1994 developed 
its Guidelines for the Use of Assistive Technology: Evaluation Referral Prescription. This was a 
major milestone for validating the provision of assistive technology services in the healthcare arena. 
These guidelines led to the inclusion of assistive technology services in the standards of the 
Commission on Accreditation of Rehabilitation Facilities (CARF), a major healthcare accreditation 
organization. As a result, the monitoring requirements for quality of care and program evaluation 
systems have necessitated many assistive technology providers in CARF accredited organizations 
to address these mandates. Numerous other initiatives at both the statewide and local levels of the 
educational service delivery sector as well as within the professional organizational level, such as 
RESNA, have begun to address the accountability and performance monitoring requirements for 
the clinical service delivery sectors. Finally, an increasing number of state and national legislative 
bills requiring accountability are being legislated. The result is that all stakeholders within the 
assistive technology community are being held to a new level of accountability and performance 
monitoring. Despite all of these efforts, there is still, however, no single measure that clearly 
satisfies all objectives and dimensions of the mandates (Smith, 1996). 

Assistive technology stakeholders of quality 

Today, the performance monitoring and accountability placed on the clinical service 
delivery system has become the norm. Unlike quality assurance, the mandates appear to be more 
than a passing fancy. While practitioners were able to drive the clinical service delivery system for 
many years, today policymakers, payers, and consumers are driving the system. As discussed 
previously, consumer pressure has been a driving force behind the performance monitoring and 
accountability agenda. The disability community has certainly been a strong force in the quality 
movement. As consumers at large have become educated, they have demanded information, which 
has forced the clinical service delivery sector to not only define quality and outcomes but to 
describe them in performance terms. Whether it is doing things right the first time or doing the 
right thing, accountability, performance monitoring, and the evaluation of outcomes has become the 
expected norm. While this has been embraced widely for many years by manufacturing, it needs to 
be fully embraced by the assistive technology community. The difficult question for the assistive 
technology clinical service delivery sector to assess and answer will be in determining whether 
what it does is indeed the right thing to do and whether it was done efficiently. What is certain is 
that the second half of this decade will be hailed as one of accountability and it will be up to society 
to determine whether the scope of assistive technology practice was provided responsibly. 
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Inherent throughout the history of the development of the scope of assistive technology 
practice and its community has been an acknowledgment that the components and entities of this 
service delivery sector are exceedingly diverse. Confounding this has been the assumption that the 
service delivery sector requires the integration of these components and entities in order to be 
successful. While this premise is debatable, the bottom line is that there has been no objective 
assessment of the assumption, albeit numerous anecdotal reports appear to support the validity of 
the premise. As a result, the practice of assistive technology has evolved into a very complicated 
service delivery model. 

Assistive technology consists of a multidisciplinary service delivery model that incorporates 
a multitude .of stakeholders. The stakeholders embody a wide range of providers and agencies that 
come from many different sectors. As such, assistive technology services may be provided in a 
variety of service delivery settings. These include healthcare, educational, academic, or vocational 
settings; a manufacturer or vendor showroom; an independent living center; a research lab; a 
personal home; a private practice setting, just to name a few. 

The array of stakeholders involved are as diverse as the service delivery settings. 
Stakeholders include: the consumer (primary and secondary), occupational and physical therapists, 
speech pathologists, rehabilitation engineers, manufacturers, vendors, payers, researchers, policy 
makers, and numerous others. It is easy to see how complicated the assistive technology service 
delivery model has become. All of this is compounded further when one realizes that in order to be 
accountable, the clinical service delivery system must respond to several different performance 
monitoring dimensions (clinical status/results; functional status; quality of life; satisfaction; and, 
cost) (DeRuyter, 1997). Each of these different dimensions in turn have varied significance to each 
of the different stakeholders, agencies, and sectors. While all stakeholders seek a successful 
outcome, not all stakeholders seek the same outcome. Therefore, what may be a successful 
outcome to one stakeholder group, may very likely be a "who cares" outcome to another 
stakeholder group. Figure I, "Perspectives of Different Stakeholders: Importance of Various 
Outcome Dimensions" lays out graphically the outcomes of interest to each stakeholder group. 
Figure II, "Perspectives of Different Stakeholders: Aspects of Assistive Technology Service 
Delivery" graphically portrays the aspects of service delivery, financial concerns, outcomes of 
interest, and research to stakeholders. 

As the stakeholders throughout the assistive technology community continue to participate 
in accountability and performance monitoring activities there will be a further delineation of the 
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roles of the various stakeholders. In the eyes of many, it is only the client/patient who is in need of 
clinical services that is considered as the customer/consumer of assistive technology goods and 
services. This is a rather myopic view, which merits reconsideration. There is a high likelihood 
that in the further delineation of roles, a majority of assistive technology stakeholders will find 
themselves not only functioning as providers of goods and services but also find themselves as 
customers/consumers of assistive technology goods and services. This will have a profound 
positive effect on the quality of service within the assistive technology community because the 
significance of accountability and performance monitoring activities depends largely on the role of 
the stakeholder. Customers/consumers measure performance largely on an intuitive basis and have 
little interest in the issues that concern or relate to the providers of goods or services. Providers, on 
the other hand, are more concerned about the quantitative and qualitative aspects of the goods and 
services. A direct consequence of the clearer role delineation should be a better ability to 
understand both sides of the clinical service delivery process that can only enhance the quality of 
the goods and services provided. In the final analysis, all stakeholders in general and the assistive 
technology community at large will benefit from accountability and performance monitoring. 
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ACCOUNTABILITY IN ASSISTIVE TECHNOLOGY INTERVENTIONS: 

MEASURING OUTCOMES 

Roger O. Smith, Ph.D., OT, FAOTA 

Hierarchy in the Complexity of Assistive Technology Outcomes 

It is as simple as needing to know whether assistive technology works. A consumer 
assumes that a device, when purchased, should work. This is particularly true when experts are 
helping to select the best devices. The simplicity, however, quickly eludes us. Very little 
information is available which empirically demonstrates the usefulness of an assistive technology 
device. Furthermore, in most situations we need to choose an assistive technology device from 
an array of assistive technology options. Therefore the question is not "Does assistive technology 
work?" but rather "Which assistive technology devices work best?" 

We also know that assistive technology is not an appliance, which when plugged in works 
by itself. Most assistive technologies require some services for selecting, obtaining, fitting, 
adapting, designing, reevaluating, repairing, and updating the technology. In fact, the federal 
"Tech Act" definition of assistive technology clearly includes assistive technology devices and 
assistive technology services. This is an important acknowledgment. For examples, 
augmentative and alternative communication devices may require vocabulary programming, a 
computer access interface may need settings to be configured properly or a seating system may 
need to be assembled with selected accessories and be individually fitted to the user. Each of 
these services can make the difference between a useful and a useless assistive technology 
device. 

Yet another dimension which can influence an assistive technology outcome is the 
environmental context. The environment for which the assistive technology is used interacts 
with the device and may match or mismatch. How well they match directly impacts the 
effectiveness of the assistive technology. How well an assistive technology system fits 
individ ual preferences, family values, cultural beliefs, and resources available in the environment 
delineates the outcome. 

Measuring the effectiveness of an assistive technology device is simple. In its context, 
however, outcome measurement multiplies in difficulty. Figure 1 highlights the increasing 
complexity of outcome questions. 
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Figure 1 



Increasing Complexity of Assistive Technology Accountability 
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What Assistive Technology Intervention Are We Measuring? 

In 1985, Barry Rodgers highlighted 19 steps in the assistive technology service delivery 
process in a paper entitled, "A future perspective on the holistic use of technology for people 
with disabilities." These 19 steps detailed the spectrum of steps required of a service provider to 
implement an assistive technology system. See Figure 2 for the complete listing. These steps 
included tasks such as identifying the need an individual has for assistive technology, securing 
the funding, evaluating the individual who may be using the assistive technology for their 
abilities and disabilities, assessing the features and alternatives available which this individual 
mi gh t use, matching the individual to the technology, ordering the technology, delivering and 
fitting the technology once it arrived, re-evaluating the individual's performance with the 
assistive technology, designing a follow-up evaluation process, and providing on-going 
maintenance to keep the assistive technology working. 

A key concept from the Rodgers' work is as important today as it was in 1985: of the 19 
Rodgers' tasks only one focuses on assistive technology device selection. The other 1 8 are 
service tasks essential to support the effective use of the device. Acknowledging the existence of 
these 1 9 components of the service delivery process is extremely relevant to outcome 
measurement. Failure in any one step in this service delivery process can result in failure of the 
assistive technology intervention. Thus, an outcome measure needs to be linked to a complete 
description of services. Otherwise, the services contributed to the outcome will not be clear. 
(From research design terminology, an outcome investigation needs to have an independent 
variable and a dependent variable). 

Major Domains of Outcome 
Two Simple Domains of Outcome 

While on one hand describing the assistive technology intervention (the independent 
variable) is more complex than it superficially appears, the quest for understanding the outcome 
variable can be described simply. Figure 3 highlights that there are essentially two domains of 
outcome. These primary dimensions are 1) the performance of an individual using assistive 
technology and 2) the cost of achieving that level of performance. Minkel (1996) highlighted 
that the value of the assistive technology is the basic formula of outcome divided by cost. Many 
other outcome measures do exit. Others, however, can be described as proxies, subsets, 
precursors or some type of contributor to these two primary dimensions. 
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Figure 2 



COMPONENTS OF A ’’HOLISTIC” DELIVERY SYSTEM 



An excerpt from "The Holistic Application of High Technology for Conversa- 
tion, Writing, and Computer Access Aid Systems." In Chapter One, Roger Smith 
refers to this paper written by Barry Rodgers which details the components of a ser- 
vice delivery system: 

"The holistic application of a high tech aid system will require the 
following components: 
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1) Locating people who can make use of the aid system. 

2) Establishing their needs and capabilities and the potential benefits of an 
aid system. 

3) Selecting and acquiring appropriate system components including special 
market hardware and software, and general market hardware and soft- 
ware. 

4) Making simple modification to hardware and software if necessary to 
make it compatible. 

5) Assembling the aid system. 

6) Mounting the aid system on the user's wheelchair, shoulder bag, bed, etc. 

7) Fitting the aid system to the user including adjustments, modifications, 
and initial customization. 

8) Selecting the most effective aid system training aids (manuals, video 
tapes, demonstration programs, etc.) 

9) Initially training the user in the basics of the system and how to 

optimize it for themselves. 

10) Training the people in the users environment who will need to help the 
user maintain the aid system. 

11) Providing ongoing training to make sure users get all possible benefit 
from the aid system. 

12) Being on call to answer subsequent questions about the aid system as it 
is being used. 

13) Providing ongoing preventive maintenance and replacement worn-out 
parts. 

14) Providing repairs. 

1 5) Updating the system when significant improvements in available 
functions make it desirable. 

16) Periodically evaluating the degree of integration of the aid system in the 
user's life and providing suggestions or further training as necessary. 

17) Using information gained from users in the refinement and improve- 
ment of the aid system. 

18) Providing a different more appropriate aid system when the user's needs 
or capabilities change. 

19) Providing a different, more appropriate aid system when significant 
advances in aid system design make more useful aid systems available. 
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Figure 3 

Two Essential Domains of Outcome 




Value = Outcome / Cost (Minkel, 1996) 
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This two dimensional model suggests that there are four potential combinations. The 
worst outcome would be an individual who has a very low performance at a high cost. The most 
positive outcome, then, would be a very high performance of an individual using assistive 
technology at a very low cost. The two mediocre combinations would be high performance at 
high cost or low performance at low cost (see Figure 4). Cost-benefit types of outcome studies 
must ultimately speak to how well an assistive technology fits into one or more of these 
quadrants. 

Types of Performance 

Personal performance, however, interacts with other dimensions that impact performance 
from the environment. Figure 5 shows the fundamental building blocks of Personal Performance 
with the addition of Intervention Performance and Support Performance blocks. Personal 
Performance includes domains of self-care functioning, such as toileting, dressing, mobility, 
ba thin g and other traditional rehabilitation activities of daily living. Personal Performance also 
includes what in the gerontological literature are called instrumental activities of daily living, 
such as community mobility, participation in community activities, and home management tasks. 
Many of the more traditional functional performance instruments from the rehabilitation 
professions fall within this category of personal performance assessment. 

Intervention Performance is of primary interest to program administrators. Program 
evaluation and quality assurance activities often examine the adequacy and effectiveness of a 
particular intervention or program. Assistive technology assessments, which collect data to 
examine the service delivery process or focus on the features of an assistive technology device, 
are types of intervention performance assessments. Data that are collected to examine durability 
of equipment, provide specifications of an assistive technology device or system, or test the 
usability of a technology examine intervention performance outcomes. 

Support Performance includes the systems around the user of assistive technology. 
Support systems facilitate the successful acquisition, integration and continuing implementation 
of the assistive technology. Availability of funding, financial services and assistive technology 
experts evaluation team, supportive family or community, vendors, maintenance facilities, 
medical services, vocational or educational services, information brokers and access to case 
managers contribute to the performance of the support structures within the service delivery 
system. 9 n 
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Figure 4 

Simplicity of Assistive Technology Outcomes 
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Figure 5 

Three Types of Performance 
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Personal Performance, Intervention Performance and Support Performance, along with the cost 
of achieving these levels of performance must operate together as a single unit. 

The Importance of Subjective Dimensions of Outcome 

In an ideal world we would be easily able to measure performance. In the real world, 
however, it is not easy to measure how an individual performs in the activities important to them. 
Likewise it is difficult to measure the performance of interventions, or support systems. 
Consequently, measurement instrumentation has received substantial attention over the last 
decade to facilitate the ease and improve the quality of collecting data. This has led to the 
development of numerous subjective measures of outcome. Subjective data instruments provide 
us with a method of collecting outcome information parallel to objective data instruments. The 
four outcome dimensions (Personal Performance, Intervention Performance, Support 
Performance and Cost) can be measured using objective or subjective data collection techniques. 
Subjective performance measures often manifest in the forms of user satisfaction surveys, self- 
satisfaction questionnaires, quality of life instruments, or preference inventories. While these 
methods of data collection are often not viewed as examining performance, at closer 
examination, these types of instruments do address performance outcomes. For example, the 
user satisfaction questionnaire, which asks an individual how, satisfied they are about a 
wheelchair and its features are a subjective measure of the assistive technology Intervention 
Performance. Or quality of life inventories such as the Sickness Impact Profile or the SF-36 
elicit the perspective of the individual on their health and other aspects of Personal Performance. 
In the supplier industry, customer satisfaction questionnaires often address aspects of 
Intervention and Support Performance. 

These types of subjective data instruments make important contributions to the outcome 
field. They provide: 

a) Important perspectives which undergird performance. For example, how an 
individual feels about the assistive technology they use has a primary impact on 
their functional Personal Performance while using the technology. 

b) Subjective questionnaires often serve as an excellent proxy for functional 
performance and other outcomes. How an individual views their personal health 
to services provided or the assistive technology devices can be a window into the 
actual performance. 
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c) Subjective data can be extremely practical to collect. Directly eliciting responses 
from an individual through a simple questionnaire can replace the need for direct 
observation or standardized assessment procedures. This is efficient. Delivering a 
questionnaire to an individual does not require professional time. Also, the quality 
of the data can be good, as individuals who are using services or devices have a 
personal investment in returning good feedback. 

d) Philosophically, it is highly appropriate to involve the end user in the data 
collection process. Subjective measurement instruments require this involvement. 

Cost seems to be one of the outcome domains, which specifically seems to be conducive 
to collecting objective data as opposed to eliciting user perspective information. However, there 
are powerful benefits in securing end-user subjective cost data. One might ask any marketing 
agency or the marketing department from any business how they determine the feasibility and 
prospect of sales of any product. They will immediately highlight that the cost of the product 
becomes a primary consideration. This cost however, is not the actual price, but the potential 
customer's perceived price of a product in the context of the customer's perceived product value. 
Secondly, obtaining the real cost of any product or service is deceptively simple. The cost of a 
product or service is not the price or charge for a device or service. The actual cost includes the 
direct and indirect expenses that accrue in the development, manufacture and delivery of any 
product or service. Keen accounting and high caliber detective work is often required to even 
approximate the real cost of any product or service. This, in fact, is one of the primary 
challenges in the economic evaluation of any health-related technology 

Figure 6 portrays the three dimensional block which includes the cost block on the 
bottom. Personal, Intervention, and Support Performance blocks on the top. Blocks representing 
objective outcome measures are in the front and parallel blocks in the back represent the 
subjective outcome measures. 

All Roads Should Lead to the Top of the Mountain 

We need to develop many successful routes to the top of the outcome mountain. We will 
encounter barricades, weather erosion, and inclines that are too steep to climb, and fatigue of our 
packhorses. However, we must remain cognizant of the reason we are blazing all these new 
routes to the top of the mountain. Fundamentally, we want to know how well an assistive 
technology device and service works to improve Personal Performance at what cost. (See Figure 
7.) This mountain metaphor, as simplistic as it seems, points out an important issue surrounding 
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Figure 7 

Many Outcome Instruments Provide Routes to Estimate 
Real Functional Performance 
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outcome measurement. Until the perfect outcome measure is found, we need to be creating new 
ways to achieve our goals of measuring outcome. 

Taxonomy of Outcomes 

What to Measure 

In 1996, during an American Occupational Therapy Association Conference, Jennifer 
Angelo and Joy Hammel held a Nominal Focus Group to collect data on the question of what is 
assistive technology outcome in occupational therapy (Angelo & Hammel, 1996). This gathering 
of occupational therapists with assistive technology expertise (76% were members of AOTA 
Technology Special Interest Section) highlighted the breadth and depth of what professionals 
often consider to be areas of outcome. Figure 8 lists the items generated in three groups from 
this group session. It can be seen that these range from global success in returning to life 
activities, to the subjective satisfaction of life activities, to the serviceability of devices, to being 
able to better use a computer. This array of types of outcomes is the perspective of only one 
profession at one point in time. The list would be even more extensive across a wider group. If 
one of our goals is to better measure outcomes, we must organize the countless types of 
outcomes. Various approaches have been used to classify outcome areas. 

The Disability Models 

Many authors, agencies and organizations have tried to provide a taxonomic framework 
around outcomes relevant to interventions offered for people with disabilities. This section 
reviews many of the prominent taxonomic approaches. These taxonomies have been generated 
and promoted from medical and health organizations, medical rehabilitation agencies and 
organizations, rehabilitation related therapy and engineering perspectives. While there are many 
similarities to these ways of looking at outcome, each has a unique flavor and contribution to 
how we view the importance of outcome and which areas are emphasized. In 1 993 the National 
Center for Medical Rehabilitation Research (NCMRR) of the National Institutes of Health 
compiled an effective overview of disability classification schemes from that time (National 
Center for Medical Rehabilitation Research, 1993). They discussed the model from the World 
Health Organization (disease, impairment, disability, and handicap) from the early 1980's, the 
model from Nagi (1991) which formulated the Disability in America model in 1991 
(pathophysiology, impairment, functional limitations, and disability), the Public Health Service 
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Figure 8* 



GROUP 1 

1. To pay income tax 

2. Ability to perform with able bodied 

3. Cost effectiveness 

4. Assistive technology within the context of other intervention strategies, 
a one-stop shopping concept 

5. Life role performance 

6. Assistive technology as a tool for consumers to meet multiple needs (such as 
when they are teaching and working) 

7. Increase educational potential for work 

8. Decrease reliance on social support systems 

9. Increase independence in daily tasks (activities of daily living, work, play) 

10. Occupational therapy’s unique role in relation to function 

11. Decrease functional impairments 

12. Community integration (this is at the handicap level according to WHO) 

13. Increase independence 

14. Occupational therapist holistic approach as optimal AT discipline 

15. Increase quality of life 

16. Identify funding barriers/solutions. Get funding for occupational therapists, AT 
and AT services 

17. Increase respect for occupational therapists role in AT among other 
professionals 

18. Achieve goal/dream beyond AT 

19. Identify barriers to achieving role performance (environment, community, 
attitudes awareness, social) 

20. Demonstrate true definition of function 

21. Recognition of OT by consumers, society, and funders 

22. Efficiency of service delivery 

23. Integration of AT teaching in professional programs 

24. Joint occupational therapy and AT certification 

25. Quality of AT/occupational therapy/AT services 

26. Consumer satisfaction with occupational therapy/AT services 

27. Having fun 

28. Assistive technology versus other alternatives 

29. Occupational therapy as At case managers 

30. Team approach 

GROUP 2 

1. Satisfaction for the client, care giver, and payor 

2. More capable of employment 

3. Decrease cost of subsequent medical care 

4. Purchase in mass market. Meaning that this will decrease the cost and therefore 
more people will be able to buy it and benefit from using it 

5. Able to meet a personal goal 
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6. Improve earning power 

7. Amount of time they use it. It would be easy to measure the amount of time that 
individuals use it. This would be one outcome measure to demonstrate that it is 
necessary because they are using it frequently. 

8. Number of additional tasks that can be performed 

9. Decrease number of hours for care taker/paraprofessional 

10. Allow independent living 

11. Elderly live at home more independently 

12. Quality of life 

13. Increase number of life roles 

14. How much faster, more efficiently can person complete task 

15. Compatible or interface with devices used at the same time 

16. Be able to participate in and graduate from school 

17. Comparison of devices/solutions 

18. Quantities of social relationship 

19. Qualities of social relationship 

20. Easy to use/user-friendly 

21. Benefit of technology justifies the cost of the equipment/services 

22. Esthetically pleasing 

23. Decrease device abandonment 

24. Serviceability 

GROUP 3 

1. Betters AT devices for increased mobility given stairs and uneven services and 
better access to those aides 

2. Did client use the AT devices that were provided? Did the client accept the 
device and was training given? 

3. How independent are they in the environment such as at work or at school 

4. Evaluate employability and employability using AT 

5. Quality of life - “seamless society” and be accepted by peers. People should not 
be set apart in society because they cannot participate in certain activities due to 
a disability. Rather all people should have access to common activities side by 
side. 

6. Good fit between person and AT device in the environment 

7. Ability to pursue own interests, roles, and goals 

8. Assessment is ongoing, of how individuals use AT and new ideas identified 
follow up 

9. Express oneself (via computers) AT 

10. Assess/implement in home environment (telemedicine) 2-way 

*Note: Items in Figure 8 are not listed in any particular order. They reflect initial 
brainstorming prior to clarification and evaluation stage of process. 
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Task Force (underlying cause, organ level, person level, and interaction of environment on 
person, family and community) and recommended the new NCMRR model which combined the 
benefits of these previous schemes. The NCMRR model continues to be promoted (Gray, 
Quatrano, & Lieberman, 1998) as it includes key features of the problems, which emerge from 
within the individual (pathophysiology, impairment, functional limitations, and disability), as 
well as highlights the interaction of societal limitation. This 1 993 scheme defines 
pathophysiology as that process which interrupt or interfere with performance on the cell and 
tissue level. Impairment is defined as the loss or atypical performance of body organs or organ 
systems. Functional limitation is defined by this NCMRR scheme as a restriction or inability to 
perform actions or activities. Disability is defined as the inability or limitations to perform tasks 
within a physical and social context. Lastly, societal limitation is defined as restriction of 
performance, which are attributable to social policy or social barriers. 

The previously mentioned disability taxonomies have several things in common. They 
tend to be presented in a linear fashion, move from a micro to a macro level, and focus on 
disability. Other taxonomies, however, break out of this linear thinking of disability. Some have 
built a larger model, which link disability toward ability, highlight interventions, and integrate 
the impact of the environment and environmental interventions on disability. 

The Balance Theory Model 

One of these models emerges out of the industrial engineering (Smith & Sainfort, 1989) 
as a human factors engineering model it focuses on the relationship between the individual and 
the technology in the environment. This model suggests that the individual and function 
performance is in the center, with four components interacting and contributing to the 
individual's work abilities. The four contributing factors are the task itself (and its demands), the 
environment (defined as the physical work environment), the organization (forces such as 
management and scheduling), and the technology available for the individual to use. This model 
places the individual as only a component of an overall work system. As this model emerges 
from engineering, however, it is not encountered in medical taxonomy discussions. 



New Institute of Medicine Model 

Recently, the Institute of Medicine (IOM) (Institutes of Medicine, 1997) updated their 
1991 model and emphasized that the earlier models focusing on disability neglect to highlight the 
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importance of the relationship of the environment to the individual. (Brandt & Pope, 1997) Thus, 
the new IOM model consequently emphasizes the critical role the environment plays on 
pathology, impairment and functional limitation. The new IOM model generalizes the concept of 
societal limitation from being an endpoint on a continuum to posing the environment as having 
an effect on all areas of: a) no disabling condition to b) pathology, c) impairment, and d) 
functional limitation. Also, disability has been removed from the linear disability continuum. 
Disability has been redefined as one possible result of the interaction between the person and the 
environment. In fact, this latest IOM model of the enabling-disabling process omits the term 
disability from the taxonomy. The state of disability is not inherent in an individual, but is a 
result of the .unique combination of the individual and the environment. This combination of 
performance leads to the ultimate quality of life. 

The HOPPIT Model 

The Human Occupational Performance Practice Integration Theory (Smith, 1998) is a 
model which emerged from the American Occupational Therapy Association uniform 
terminology from 1979, 1989 and 1994. (American Occupational Therapy Program, 1989, 1994) 
The HOPPIT Model extends the AOTA uniform terminology and many of its concepts have been 
integrated into a functional assessment software package called OT (Smith, 1990, 1992 & 1994, 
1995, 1998) OT FACT has been presented as one means for investigating the outcomes of 
assistive (Davel & Smith, 1996). 

The relationship of outcome assessment areas according to the HOPPIT Model is 
attached as Figure 9. As can be seen, at the center of the model is the individual which includes 
four levels of functional performance. On the bottom level resides the underlying Components of 
Performance which include functional categories such as range of motion, pain, strength and 
memory. Above that lies the Integrated Skills of Performance that include categories such as 
hand function, problem solving and work tolerance. Above that are the Activities of 
Performance which include the traditional self-care activities, as well as traditional IADL 
vocational and educational role-related activities. On the top sits the Integrated Roles of 
Performance that include the various types of roles an individual might need to perform. Each of 
these levels of performance is dependent on all of the categories from the levels below. 

To the side of the individual performer resides the Environment, which includes social 
support systems, financial support systems, orthotics, assistive technology and other 
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Figure 9 

Relationship of Outcome Assessment Areas 
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(Adapted from Smith 1991, 1995, 1998) 
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environmental factors. These environmental categories can affect any level of individual 
performance. 

On the other side of the individual performance resides therapy intervention. This 
includes interventions such as sensory integration which focuses on the Underlying Components 
of Performance, to ADL training which focuses on the Activities of Performance, to self range of 
motion which focuses on the Activity of Performance, work hardening which would focus on the 
integrated skills of performance. Therapeutic intervention can interact with the Environment. 

This is a comprehensive model of outcome, so the person's overall performance is 
dependent on the full combination of the environment, the individual and the therapeutic 
interventions.. 

Also advocated by the HOPPIT Model are the six approaches of intervention. Figure 10 
highlights six approaches of intervention. These are categorized into three types: a) interventions 
which are due to performance changes of the individual (environment-free), b) approaches which 
are adjustments in the environment (environment-adjusted), and c) an approach which is 
dependent on another person (environment-assisted). The example in this figure is for an 
individual who is blind and is attending a lecture where slides are being shown. Approach #1 is 
to reduce the impairment, which would require the rehabilitation of vision so the individual can 
see the slides. Approach #2 would be to build compensatory skills, such as having the individual 
learn techniques such as asking questions to clarify the information during the lecture. Both 
Approach #1 and #2 are environment-free approaches. Approaches #3, 4 and 5 fall within the 
environmental-adjusted type. Approach #3 is to use assistive technologies such as obtaining an 
electronic copy of the overheads ahead of time and using a headset with a speech synthesizer. 
Approach #4 is to change the task. For example, the instructor could verbally describe the 
contents of the slides during the lecture. Approach #5 is to change the task expectations, such as 
making it okay to not read the slides. Perhaps reading the textbooks more carefully ahead of time 
would make the slides moot in terms of information. Lastly, Approach #6 is to use person 
assistance, such as having another student fill in any information gaps during or following the 
lecture. 

The mechanism used by the HOPPIT model to differentiate between environment-free, 
environment-adjusted and environment-assisted data collection facilitates the ability to 
discriminate the impact of an assistive technology intervention from other interventions. For 
example, as in Figure 1 1, we could plot an individual's functional performance over time using 
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Figure 10 

6 Approaches of Intervention 
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based on concepts discussed in Smith, 1994 



Functional Performance 



Figure 11 

Environment-Free and Environment- Adjusted 
Performance Over Time 




Environment-Free Environment-Adjusted 

= Area indicating impact of assistive technology 
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an environment-free assessment. This would determine the performance of an individual with no 
environmental or outside assistance and monitor their progress. At the same time we could 
collect data using an environment-adjusted data collection format where assistive technology is 
provided to the individual and performance is ascertained with this environment-adjusted 
condition (Davel & Smith, 1996). Under this scenario, the area on the graph between the 
environment-free and environment-adjusted lines is the outcome specific to the assistive 
technology intervention. 



Outcome Measures for Assistive Technology 
Traditional Instruments 

Volumes II and III of this RESNA resource guide identify the interest of the assistive 
technology field as it tries to define outcome measures, validate new methods for collecting 
reliable and valid data which can be used to examine assistive technology outcome. Ideally the 
field needs outcome measurement instruments to work across populations, settings, functional 
performance areas, assistive technology devices and assistive technology services. It would seem 
that the substantial attention and research in medical rehabilitation outcome assessment, some 
measures would have been developed which are appropriate. Typical rehabilitation functional 
assessment measures, however, even those that have become the foundations of rehabilitation 
outcomes research, are not designed from an assistive technology perspective. Stineman 
(Stineman, 1998) pointed out that the FIM (functional independence measure) and the SIP 
(sickness impact profile) are important as they collect data across a spectrum of populations. 
However, neither were created considering assistive technology, and in fact are missing essential 
components relevant to assistive technology. For example, in the FIM assistive technology is 
embedded into the seven-point scale so it cannot tease out the impact of assistive technology. 
Furthermore, the FIM does not allow the acquisition of a total score when an individual is using 
assistive technology, even though they may be as independent as an individual who is not using 
assistive technology (this is the case with all functional assessment measure which use a medical 
model designed from an environment-free perspective). 

Relevance of the SF-36 

The SF-36 instrument (Ware & Sherboume, 1992) is relevant in the discussion of 
assistive technology outcome for three reasons. First, the SF-36 represents quality of life 
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instruments developed primarily to assess medical technology outcomes. These quality of life 
instruments are being used prolifically and having a tremendous impact on the financing and 
delivery of health-related technology. While many of these instruments are not designed to 
measure the impact of assistive technology, we have much to learn from these instruments in 
how they are applied, in how they uniquely collect quality of life data. 

Secondly, the SF-36 is an instrument which is probably the most widely used in the world 
for measuring medical outcome. Hundreds of well-cited studies have used the SF-36 as a key 
data collection instrument. In fact, a volume of bibliographic entries documenting of SF-36 
related studies is available. The development team of the SF-36 has also embarked on a major 
international project translating the SF-36 and validating the instrument for cultures and 
languages across the globe. As a part of the internationalization of the SF-36, very interesting 
discoveries are being made. One of these is a discovery of how sensitive the SF-36 is for 
discriminating quality of life differences among large populations. Ware (1997) pointed out at the 
International Congress on Performance Measurement and Improvement in Health Care that the 
translation of the SF-36 from American to British English uncovered an unexpected finding. 

One of the questions in the SF-36 asks the individual about how they would do in walking a 
mile. As the British system uses metric as opposed to the English mile, the SF-36 translation 
team converted the word mile to kilometer. While they understood that the kilometer was not 
exactly equal to a mile, both were long distances and it seemed would be functionally equivalent. 
What resulted was that the British group encountered the kilometer to be easier to walk than the 
mile. The SF-36 in its relatively few number of questions which target global health-related 
questions, discriminated subtlely between one long distance and another long distance. 

Third, investigations using the SF-36 are finding perplexing results which may relate to 
interventions such as assistive technology. A team from Italy found substantial differences in 
quality of life between northern and southern Italy. They thought that these differences might be 
explained by the demographic differences. So, they examined age, medical case mix, gender, and 
a variety of socioeconomic factors. None of the large number of demographic variables 
scrutinized explained the differences between the perceived quality of life differences between 
northern and southern Italy (Apolone, 1997). The phenomenon of unexplained differences 
between populations is relatively common within the SF-36 data. Ware has clearly stated the 
widespread interest in identifying causes of these differences (1997). 

The significance to assistive technology is that when asked, authors of the Italian study 
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and Ware agreed that rehabilitative interventions such as occupational and physical therapy and 
the use of assistive technology might be plausible explanations for these quality of life 
differences. Rehabilitation intervention variables, however, are not identified. In fact, data 
pertaining to these interventions is not collected in most health technology assessment 
investigations. The SF-36 research teams agreed that further investigation along this line is well 
warranted. 

While the SF-36 is an extremely global measure of quality of life, when used in large 
numbers it discriminates well between interventions. It may be prudent for assistive technology 
outcomes research to consider the SF-36 as an additional data collection platform. 



O 

ERIC 



Economic Evaluation of Assistive Technology Interventions 
The QALY 

The logical extension of quality of life outcomes is to examine the cost compared to 
benefits of an intervention. Oldridge reviewed the health-related quality of life strategies and 
economic evaluation methods, suggesting the relevance to assistive technology outcome 
measurement (Oldridge, 1996a; 1996b) Measurement methods such as the time trade-off 
technique, the basic lottery ticket and the thermometer are useful for comparing the qualities of 
possible outcomes of an intervention. Also, for comparing interventions these quality of life 
results can be integrated into an economic formula creating "quality adjusted life years", also 
known as QALY's. 

There is some dissension regarding the appropriateness of using QALY's to measure 
outcomes of people with disabilities (Wolfensberger, 1994). There also remain issues pertaining 
to the reliability of QALY's and the subjective nature of its preferential-type data. For example, 
people with different perspectives will weight the outcomes differently (Richardson & Nord, 
1997). Nevertheless, this type of data collection based on user opinion is gaining not only 
attention, but also increased usage in practice. The health technology assessment and health 
systems analysis literature documents numerous studies using decision analysis methods such as 
QALY’s. 

To date, few economic evaluations have been performed which examine the cost 
comparing benefits related to assistive technology (see cost effectiveness, cost efficacy and cost 
utility in the earlier chapter by DuRuyter). Applications and adaptations of these methods, 
however, deserve careful scrutiny as outcomes resemble methodologies addressing assistive 
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technology. One research track in Europe has taken an international team to examine the cost- 
benefit of assistive technology. While these studies are preliminary, this is clearly new territory 
foraged by this team. 

The CERTAIN Project 

The CERTAIN Project (Cost Effective Rehabilitation Technology through Appropriate 
Indicators) was orchestrated by the Technology Initiative for Disabled and Elderly (TIDE) as a 
program of the European Union. This particular project, based in Italy, investigated the 
feasibility using mainstream health care technology assessment measures, a new cost-outcome 
method, and a Quality of Adjusted Life Years approach for application in assistive technology 
(Andrich, Ferrario, & Moi, 1998). The methodology was a retrospective study of seven cases 
representing a wide range of ages, diagnoses, levels of disability, and types of assistive 
technology used. 

The research team considered five domains of outcome. These were 1) Context of 
outcome (e.g., culture, demographics, and experiences), 2) Individual goal expectation (including 
individual performance and social relationships), 3) Family expectations, 4) Professional s 
expectations, and 5) Outcomes with community activity. Costs were extensively examined. 
Careful scrutiny was taken to collect both assistive technology device and assistive technology 
service cost data across both direct and indirect expenditures. 

Several measure of quality of life and functional performance outcome were used, 
including the FIM, EuroQol, Index of Health-Related Quality of Life and the MacMaster Health 
Classification System. Some adaptations of these instruments were necessary to better focus on 
assistive technology. 

The CERTAIN Project revealed several findings of significance to assistive technology 
outcomes research. Regarding the FIM, it was found to report the performance outcome of 
assistive technology in an uneven way. When an assistive technology device was a direct support 
specific to a FIM area of function it was sensitive. However, when an assistive technology 
device addressed functions outside of FIM categories or implemented environmental adaptations, 
the FIM did not exhibit the impact of the assistive technology. Regarding the health quality of 
life assessments which the project used as utility scales, there were five major results (Ferrario, 
1996). First, the generic quality of life measures used ineffective terminology. For example, 
questions reflected expectations that the individual was walking. These questions must be re- 



Volume I: RESNA Resource Guide for Assistive Technology Outcomes: Measurement Tools 



Page 39 



phrased when used with people with disabilities. Second, the weighting of the scales used in the 
traditional scaling of the quality of life instruments reduced the changes detected to almost zero. 
Again customization of these general quality of life scales is needed to highlight smaller, yet 
significant, quality of life improvements. Third, calculation of QALY's showed that with 
progressive disorders the long-term benefit of the assistive technology was subsumed by 
changing needs. Traditional quality of life instruments assume some stability. Often in real 
applications, however, an individual's state changes and therefore do the benefits and updated 
needs for the assistive technology. A downward "decay" or upward "habilitation" factor needs to 
be built into a disability quality of life model. Fourth, the timing of the quality of life changes for 
assistive technology users is not consistent and thwarts standard fixed time data collection 
intervals commonly used in general health assessment methodology. Acceptance of an assistive 
technology device, timing and appropriateness of assistive technology services, and learning 
curves for integrating sophisticated assistive technology into daily life all affect quality of life 
outcome and must be built into a quality of life model which targets people with disabilities. 
Lastly, orthodox quality of life instruments do not include the family or other primary supports. 
This dimension needs to be incorporated into quality of life. 

Overall, the CERTAIN Project cuts a new path into the cost/outcome measurement 
methodology in assistive technology. This project demonstrated the relevance, promise and 
pitfalls of traditional health-technology assessment methods and instruments. These types of 
investigations are needed to move assistive technology outcome measurement forward. A new 
generation of instruments and methods is one the horizon. Perhaps the CERTAIN Project can 
launch the interest, activity, and support for this important undertaking. 
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PROGRAM EVALUATION METHODS 



Jan Galvin 
Introduction 

The structure of the U.S. health care system is changing rapidly, primarily in response to 
concerns about increased costs of health-related services. Many of these changes appear to 
challenge our view of the delivery of quality services. Expanding the information available on 
quality requires the development of valid measurement tools and routine access to the right data. 
The Institute of Medicine (IOM) has defined quality as “the degree to which health services for 
individuals and populations increase the likelihood of desired health outcomes and are consistent 
with current professional knowledge.” This definition suggests that a.) Quality performance occurs 
on a continuum, theoretically ranging from the unacceptable to excellent; b.) The focus is on 
services provided by the delivery system; c.) Quality may be evaluated from the perspective of 
stakeholders; d.) Research evidence must be used to identify the services that improve health- 
related outcomes; and, e.) In the absence of scientific evidence regarding effectiveness, professional 
consensus can be used to develop criteria. This definition of quality illustrates the complexity of the 
concept and its evaluation. However, it also suggests that despite this complexity and the current 
dynamics of the health-care delivery system, much can be done to: identify and balance competing 
perspectives, establish a framework with explicit criteria by which services can be judged, and, 
facilitate appropriate information systems for monitoring and reporting. (McGlynn, 1997; Landrum, 
et al, 1995) Program evaluation is one tool that, if developed with care and skill, can be used in 
conjunction with outcomes measurement and management, and a commitment to continuous 
quality improvement, to provide quality, competitive services that meet consumer needs to their 

satisfaction. 

Program Evaluation 

Programs need to show that they are delivering what they promise. Program evaluation 
refers to the variety of information-gathering activities designed to assist in program 
development, function and ongoing improvement and show program managers, the organization 
and stakeholders what they are delivering. 

Program evaluation has been utilized in social services since the late 1960s. In the early 
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seventies, education and health care began implementing program evaluation systems followed by 
acute medical rehabilitation in the late 1970s. Methods for determining what is evaluated and how 
it is to be evaluated have been refined over the past 20 years as have the methods of collection, 
moving from abstracting from narrative records on paper to highly automated pooled, multi- 
institutional databases. (Wilkerson & Johnston, 1997) Program evaluation is now considered a 
solid analytical approach to facilitating internal review of an organization's programs' effectiveness 
and efficiency, and to build evidence for external communications about the programs. (Wilkerson, 
in press) As the Assistive Technology field moves into developing outcomes measurement systems, 
we should look carefully at what has been developed in the acute medical rehabilitation field and 
choose judiciously from their hard work. 

The concepts of outcomes measurement, outcomes management, program evaluation, and 
quality improvement are closely related, and the interchange of terms is often confusing. 

(Forer, 1996) 

Outcomes Measurement. The degree to which the program produces sustained practical 
improvement in patient function. (Johnston, Wilkerson, Maney, 1993) 

Outcomes Management. "A technology of patient experience designed to help patients, payers, 
and providers make rational medical care-related choices based on better insight into the effect of 

these choices on the patient’s life". (Ellwood, 1988) 

Program Evaluation System. A system defined by customer and stakeholder needs that includes 
process evaluation, program efficiency, program effectiveness, impact, and outcome evaluation. A 
program eval uati on system provides information an organization can use to improve performance 

to an agreed-on level that meets desired outcomes. 

Quality Improvement. A system through which better and more efficient processes in an 
organization continuously raise the levels of performance and the outcomes of all functions, 
eliminating errors, and reduce costs. (MacDonnell, 1996) 

What is Program Evaluation? 

Traditional evaluation literature (Knutson, 1969; Weiss, 1972) cites two major types of 
evaluation. First, formative evaluation is used in an ongoing program and serves to identify needed 
corrections and improvements in the service and/or program. A second model of program 
evaluation summative evaluation is used to determine whether a completed program was 
successful. However, the most important function of a program evaluation system is that it becomes 
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a continuous process of assessing client and program needs and objectives, and uses that 
information to improve the individual’s and the program’ performance. For this to occur, the 
organization must buy into program evaluation and outcomes management and must be committed 
to change. 

CARF 1 has defined program evaluation as a “systematic procedure for determining the 
effectiveness and efficiency with which results are achieved by persons served during service 
delivery and following program completion, as well as the individual’s satisfaction with them.” 
Program evaluation management reports tell the facility/provider whether its performance is 
acceptable according to the goals, objectives, and expectancy levels that the facility/provider has 
established for itself. (CARF, 1995) 

JCAHO 2 specifies a process that involves four functions. 

1. Identifying the most important aspects of care and service an organization, a department, or a 
program provides; 

2. Using measurable indicators to systematically monitor these aspects of care and service in an 
ongoing way; 

3. Evaluating the care and service, when predetermined thresholds are reached, to identify 
opportunities to improve the quality of car and service; 

4. Taking actions to improve care and service or to solve problems, and evaluating the 

effectiveness of these actions. (JCAHO, 1992) 

Program evaluation involves setting goals and expectancies. If goals are not attained, action 
should be taken to determine why. Program evaluation systems should be used to help make 
decisions about programs and as a tool for program improvement. (Johnston, Wilkerson, Maney, 
1993). A program evaluation system can be very simple and inexpensive or very complex and 
computerized. Whichever framework is chosen for program evaluation the system must be able to 
measure the efficacy of services, the effectiveness of services and program, and the efficiency of 
services provided. (Wilkerson, in press) Regardless of the organizations’ size and complexity there 
are a number of basic elements that must be present if the organization is to measure and manage 
outcomes successfully. These are: 
mission statement 

A mission statement should reflect the essential reason that an organization exists usually expressed 




1 CARF... The Rehabilitation Accreditation Commission 

2 JCAHO The Joint Commission on Accreditation of Healthcare Organizations 
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in one sentence. A mission statement helps people outside the organization gain a general 
understanding of the organization’s mission and the needs that it is meeting. It also helps those 
developing a program to really narrow down the focus of their services or program. 

• program structure 

The organizational chart reflects where the program is in terms of administration and relationship to 
other programs and services of the organization. 

* program goal statements 

Perhaps one of the most important elements of all. These statements, if well thought out can 
significantly help a program determine what outcomes are really important, what can be measured 
and how. If not well-thought out, they are unusable, probably unmeasurable and will lead to a 
program that has no defined goals and one that will be unable to defend itself. Involving persons 
served and payers, as well as program staff in the development of goal statements will help reflect 
their values. Through related measures, persons served and payers will be able to determine if then- 
expectations have been met. 

* admission criteria 

The admission criteria detail characteristics, behaviors, disabilities, or other qualifications that a 
person must possess to be admitted to the program. Be very clear and specific in justifying these 
criteria. It will also then, serve to provide objective reasons for non-admission. Clear concise 
admissions criteria will assist third-part payers and other referral sources in directing clients to the 
most appropriate resource. 

• list of persons served 

This list clarifies and justifies the nature of the persons whom the organization expects to benefit 
from services. This list should show a logical relationship between persons served and the program 
goal statements, and between persons served and admissions criteria. 

• list of services provided 

This is a list of the services the program offers. It also lists the resources available for an individual 
to achieve his or her goals. There should be a logical relationship between services provided, 
persons served, and program goal statements. 

* statement of objectives 

This is a statement that reflects the specific Quantifiable objectives to meet the program goals. The 
statement of objectives should clearly express the desired changes expected following the provision 
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measures 



Measures are methods an organization uses to indicate progress toward objectives. They should 
include efficiency measures, effectiveness measures, and measures of customer satisfaction. 

• time of application of measures 

Time of application refers to when measurement takes place, such as at discharge, one week later, 
etc. 

• to whom measures will be applied 

This specifies clearly who will be measured. In most evaluation systems all measures are not 
applied to all patients. An example would be: all clients returning to work after traumatic injury. 

• person served descriptors 

Person served descriptors define the persons the program is serving in terms of severity of problem 
or barriers to success. An example would be: percentage of persons served with traumatic brain 
injury; percentage of persons served who are ventilator-dependent. 

• performance expectancies 

This reflects the degree to which a program expects to achieve an objective. An example would be: 
Measure persons returning to work after discharge. Expectancy 75%; Actual result 82%. 
Performance expectancies should also be tracking costs, both expected and actual. This type of 
comparison of expectancies and actual outcomes provides a range of expectancies and helps put 
actual results in perspective, providing the organization with data by which to judge the program. 

• statement of the relative importance of objectives 

All objectives must be weighted, as they are not all equally important. A weighted system should be 
designed to give greater weight to objectives that contribute to the overall program goals. 
(Wilkerson, in-press; MacDonell, Wilkerson, Acquaviva 1996; CARF, 1995) 

A well-planned and implemented program evaluation system that includes these basic 
elements will answer six very important questions. These are: 

1 . Have we set the right goals? (goals, objectives and measures consistent with program mission) 

2. Are we reaching the goals we have set? (goals are measurable, performance is measured, 
performance to goal comparisons are tracked) 

3. How valuable is our program? (the comparison of effectiveness to efficiency, to consumer 
satisfaction) 

4. Are consumers satisfied? (Satisfaction with services provided; degree to which consumer 
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was involved in planning, etc.) 

Where do we need to improve? (tracking performance against goals or targets, reviewing 
and utilizing data for continuous quality improvement, CQI) 

6. How do we compare? (The programs effectiveness and efficiency in comparison to other 
programs). (Wilkerson, in-press;) 

Conclusion 

In these days of cost cutting by facilities and reductions in payments for services, a well- 
planned program evaluation system can provide solid information for organization wide decision 
making. It is also increasingly important to consumers, payers and providers to know how a 
programs effectiveness and efficiency compare to other programs. As we begin to develop program 
evaluation systems for Assistive Technology we need to focus on ways to make the information we 
collect useful to all stakeholders and look for ways to utilize the information to enhance quality of 
services. 
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CHARACTERISTICS OF A MEANINGFUL OUTCOME ASSESSMENT 

Marcia J. Scherer, Ph.D. 



In a 1992 Rehab Brief, the National Institute on Disability and Rehabilitation Research 
(NIDRR) summarized the fundamental role of measurement, assessment, and evaluation in 
rehabilitation as follows: 

Consumers are measured to establish their eligibility for benefits or services, to determine 
which services are appropriate, to assess their needs, to ascertain their current level of 
functioning, and to estimate their potential (NIDRR, 1 992, p. 1). 

Regarding outcomes measurement, the goal is to measure and document the effectiveness of 
AT as an intervention. In other words, to show that AT does, indeed, make a difference for those 
who use AT. 

The development of any outcome assessment instrument or measure requires field testing 
and careful analyses of the results before use can be expanded. Before purchasing or using an 
outcome assessment instrument, it is a good idea to make sure it has gone through a rigorous 
process of development and validation and that a manual exists to guide you through its correct use, 
interpretation, etc. The traditional approach to instrument development and validation is as 

follows: 

1. Domain definition 

The behaviors and traits of measurement interest will be identified. For example, is the 
measurement interest in global physical functioning or on something more specific such as 
mobility. Identified traits and behaviors will form an item pool of potential questions to 
include in the assessment instruments. 

2. Content validation 

A pool of items are generated and each is judged by its 

* clarity 

* importance 

Then acceptable items are assembled into a pilot version of the measure. 

3. One-to-one administration 

The items will be administered to a few individuals representative of the population for 
whom the instruments are intended. Draft instruments will be distributed after obtaining 

5 8 

Volume I: RESNA Resource Guide for Assistive Technology Outcomes: Measurement Tools Page 50 



informed consent from all participants. After participants in the one-to-one administration 
have completed the draft instruments, items will be discussed with each individual to check 
for any problems with wording, cultural insensitivity, directions, or level of difficulty. 
Based on the results of the one-to-one administration, revisions to the instruments and their 
administration will be made. 

4. Field testing and analyses 

Larger numbers of individuals will participate in the process and complete the assessment 
instruments. Item analyses will be performed as will factor analyses on the instruments' 
subscales when a sufficient sample of responses has been obtained. Item analyses will 
measure item difficulty and variability and an estimate of the reliability of the instrument. 
The mean and standard deviation of the item traditionally estimate item difficulty and 
variability. A common method of determining the item's discrimination is to correlate the 
item with the total score. This correlation is called an item-total correlation. Factor 
analyses on the subscales will be viewed as indicative rather than definitive when working 
with small samples. 

5. Revisions 

Items will be retained, modified, or deleted based on the results of the item analyses. It may 
take several years before a final version of an assessment instrument is ready for wider 
distribution. 

6. Measure validation 

To be considered a quality measure, an assessment instrument should provide evidence of 
its validity and reliability. 

VALIDITY refers to the extent to which an assessment instrument measures what it 

purports to measure. It is an indication of the instrument's usefulness and appropriateness. A 

measure may be valid for use in one setting or with one population, but not others. There are 

several different kinds of validity, examples of which are listed below. 

CONTENT VALIDITY is the extent to which an instrument's items derive meaningful 
information according to judgments by experts (professionals in the field, expert users, etc.). 
When experts read the items and judge that each one taps into a meaningful area, it is said that 
the measure has face validity. 

CRITERION-REFERENCED VALIDITY is not based on expert opinion, but rather how 
well it measures a particular outcome. 
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CONCURRENT VALIDITY focuses on a sample of behavior as indicative of actual behavior 
(e.g., how far one can walk in the clinic is the same as how far one can walk at home) or how 
well a measure correlates with other similar measures. 

PREDICTIVE VALIDITY is concerned with forecasting a future behavior (e.g. FIM scores 
predicting hours of required assistance). One type of validity that deserves more attention in our 
field is consequential validity, or the degree to which the results are used in a manner beneficial 
to the system, profession or field. 

CONSTRUCT VALIDITY is the degree to which an assessment instrument measures its 
underlying theoretical construct. For example, IQ is a theoretical construct that is not directly 
observable, so IQ tests were designed to try and quantify (or measure) it. Convergent validity is 
concerned with how well the results are correlated with other indicators of the same behavior 
e.g., other IQ test results). 

DISCRIMINANT VALIDITY addresses how well the measure remains uncontaminated by 
variables unrelated to the behaviors of interest. For example, if you want to assess the capacity 
for independent ambulation, items that really assess energy level or fatigue need to be examined 
for retention. 

SCREENING MEASURES try to approximate more comprehensive clinical evaluations or 
professional judgments. Their validity is assessed according to how well they accurately pick up 
those individuals with the condition and also how well they screen out those who do not have 
the condition. (See Figure 1 ). 



RELIABILITY of measures is concerned with "reproducability" of results or the degree to 
which a measure yields the same results when administered across different points in time (test- 
retest reliability) or by different raters (inter-rater reliability) 



7. Finalization of instruments and process 

Instruments are traditionally considered final versions when all field and pilot testing is 
completed and reliable and valid instruments exist with a sufficient number (minimum 
number for "minimum data sets") of quality items. In the field of rehabilitation, however, 
finalization involves ensuring accessible formats and the accommodation of persons 
regardless of age, gender, race, national origin, color, or disability. 
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FIGURE I 

VALIDITY OF SCREENING TESTS 



SENSITIVITY 

Correct Identification of those with 
(hearing loss) 

A sensitivity of 94% = 

94/100 of the test administrations 
showed hearing loss in persons with 
hearing loss 

6/100 of the test admnistrations 
identified hearing loss in persons 
without loss 



SPECIFICITY 

Saying someone does not have 
a (hearing loss) when in fact, they don’t 

A specificity of 72% = 

72/100 of the test administrations indicated 
no hearing loss in persons with no loss 

28/100 of the test administrations indicated 
no hearing loss for persons with loss 



TO CALCULATE THE SENSITIVITY AND SPECIFICITY OF A SCREENING TEST: 

1. Set up a 2x2 table as follows: 

Condition 

Present Absent Total 



Positive 


50 


15 


65 


Negative 


10 


405 


415 


Total 


60 


420 


480 



2. Sensitivity = 50/60 x 100 = 83.3% 

3. Specificity = 405/420 x 100 = 96.4% 

4. Percent of false positives = 15/420 x 100 = 3.6% 

5. Percent of false negatives = 10/60 x 100 = 16.7% 

6. Prevalence of condition = 60/480 x 100 = 12.5% 

7. Predictive value of a positive test = 50/65 x 100 = 76.9% 

8. Predictive value of a negative test = 405/415 x 100 = 97.6% 

c. M. J. Scherer 
2/13/90 
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SELECTING AND USING MEASURES REVIEWED IN THIS BOOK 

Obtaining the most useful information for AT decision-making requires a combination of 
consumer observations and input, subjective professional judgment, and objective measurement. 
When you decide to use a measure for assessing a particular domain, it is often helpful to have 
some guidance in asking the most appropriate questions and addressing key assessment or 
measurement issues. 

The following has been adapted from the book, "Psychological Assessment in Medical 
Rehabilitation" (Cushman & Scherer, 1995) for the purposes of outcomes measurement in AT. 

I. Determine the Information Needed/Desired 

There are many measures reviewed in this book. Where do you begin in selecting among them? 
The following are some key questions to ask at the beginning of the assessment process: 

1 . What is already known about the domain(s) of interest? 

2. Who can best provide the desired data? 

3. Whose perspective is it important to obtain? 

4. What is important to find out now in these areas to best help the individual and guide AT 
decisions? 

5. What will be important to know for later AT decisions? 

Regarding the use of measures for outcomes assessment, examples of key questions to consider 

are: 

1 . Who wants to know information on outcome achievement? 

2. What is expected/acceptable as evidence of outcome achievement? From whose perspective? 



ERIC 



II. Specify what is to be Measured 

The assessment of core capacities, functioning/performance, and outcome require different 
measures as indicated by the variety of measures reviewed in this book. Depending on the focus at a 
particular point in time, it may be less important to measure, for example, ambulation than muscle 
tone. Examples of key questions to consider are: 

1 . What information is wanted from a measure? 

2. Is the desired information specific or general? 

3. Given the information needed/desired, which measure(s) in this book addresses the area(s) in 
need of assessment? 

4 Can the desired information be measured accurately and with precision? 
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5. Will measurement yield better information than professional judgment? 

III. Identify what Exists to Systematically Collect the Desired Information/Data 

Once the assessment goal is established, this book reviews a variety of measures some of 
which are very specific, some global; some are unidimensional, others multidimensional. Some 
require considerable time to administer, others are brief and can be completed in a few minutes. 
Some require paper-and-pencil, others a computer. Knowing what information is desired and 
weighing that against the available time and resources to obtain it will help narrow the possibilities. 

It is becoming increasingly common to turn to brief, abbreviated, measures in order to save 
time and costs. The use of brief measures for the assessment of complex problems and for reaching 
complex treatment/rehabilitation/ outcome decisions has frequently left important information gaps. 
The results may not consider consumer background, ethnic/cultural and gender differences, etc. 
which can affect the validity or meaningfulness of the results regarding that individual. Some 
measures, however, assess a selected minimum number of behavioral indicators to be determined 
"sufficient." Thus, there are trade-offs to be weighed between the use of abbreviated and long 

versions of measures. 

Examples of key questions to consider are: 

1 . Is a measure the best way to obtain the information needed/desired? 

2. Do you want/need to construct your own measure? If so, the procedures and ethics of 
instrument development need to be consulted. 

3. Which measure described in this book seems the most applicable for this individual? 

4. Is a global form most appropriate, or one with subscales? 

5. Is there an abbreviated version? What information is sacrificed by using it? 

6. What resources are required to use this measure (cost of materials, time to administer and score, 
computer use, etc.)? 

7. Are q uali fied personnel available to administer, score and interpret the results of the measure? 

IV. Determine who Needs to be Involved 

Beyond the involvement of the consumer, it may be useful to assess family members and 
significant others as they may be the ones the consumer needs to rely upon to maintain 
rehabilitation outcomes. Persons with disabilities often require help in completing measures. They 
may need to have questions read to them and/or have someone fill in the answer sheet. There may 



need to be a sign language or other langu ranslator present. The selection of an assistant needs 
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to be made carefully in order to obtain honest answers on the measure. Examples of key questions 
to consider are: 

1 . Who can best provide the information needed/desired? 

2. Does the respondent need another person to assist him or her? 

3. Will the assistant interfere in any way with the collection of accurate information? 



V. Select Measures/Obtain Informed Consent 

The selection of an assessment instrument requires attention to its psychometric properties 
and a weighing of the risks and benefits to the consumer. "Informed consent" is defined in the 1985 
Standards for Educational and Psychological Testing (presently under revision). It is important to 
realize that the broader the domain assessed, (from ambulation to ratings of social behavior) there 
will be increasing losses of clarity and specificity. Acceptable levels of subjectivity should be 
considered. Examples of key questions to consider are: 

1 . Does the instrument describe, screen, predict or evaluate? 

2. What are the populations to which this measure applies? 

3. Is the reported psychometric evidence adequate? 

4. Will the measure yield useful information regarding this individual and AT? 

5. What are its limitations? 

6. Does it have utility in an outcome-based health care system? 

7. Does it meet the Standards of the Code of Fair Testing Practices? 

8. Are the methods and means of obtaining scores/profiles worth the time and expense to get 
them? 

9. What are the risks and benefits to the individual? 

10. Are qualified persons available to administer, score and interpret the results? 



VI. Administer Measure(s)/Consider Adaptations, Modifications, Accommodations 

It is often necessary to make modifications and adaptations in measures and assessment 
situations to accommodate individuals in rehabilitation settings. Limitations in attention span, 
memory and other cognitive functions as well as the capabilities to hold a pencil, use a keyboard, or 
see print may require adjustments in the assessment process. Examples of accommodations for 
persons with hearing loss are the use of sign language interpreters and more graphic assessment 
measures. Persons with vision loss can benefit from large print, braille or audiocassette versions of 
instruments. Individuals with motor impairments may require such accommodations as 
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computerized assessment or the recording of responses by an assistant. Suffice it to say here that 
AT specialists must be a model for all other professions in accommodating the specialized needs of 
individuals with disabilities in all assessment situations. AT needs to also be a role model for the 
involvement of consumers in the assessment process. 

Nearly all assessment today occurs in a multicultural context, also necessitating adaptations 
in measures and assessment. Yet, cultural loadings (the effects of cultural experience) can be either 
desirable or undesirable depending on the construct being assessed, the context, and the inferences 
to be drawn. The assessment of individuals from culturally diverse populations should include only 
measures that are culturally and racially bias-free and which attend to such cultural sensitivities as 
variations in definitions of terms, views of confidentiality, and cultural differences toward self- 
disclosure. The language used in the measure should be understandable to the individual with any 
culturally offensive or controversial material removed. If the measure selected is not available in 
the language of the particular consumer, another measure will need to be administered or the results 
are apt to be meaningless, or worse, misleading when either false positive or false negative results 
are obtained. Other issues regarding the assessment of persons from diverse cultural populations are 
discussed in Guidelines for Providers of Psychological Services to Ethnic, Linguistic, and 
Culturally Diverse Populations (APA Office of Ethnic Minority Affairs, 1993). Examples of key 
questions to consider are: 

1. Will the individual understand all the words and phrases in the measure's items and not be 
offended by any of them? 

2. Do accommodations need to be made in the measure or assessment process to get accurate, 
unbiased, and meaningful results? 

3. Will the individual interpret items in the way intended by the developers of the measure and by 
providers? 

VII. Score Measures 

Scoring can be time-consuming and expensive. If hand scored, the process can be slow; 
computer scoring requires access to a computer or the requisite funds to pay to have it scored. It is 
often more efficient to train support personnel to score the more straightforward measures. 
Examples of key questions to consider are: 

1 . Do the necessary resources exist to properly score the instrument? 

2. Who will get the results? 

3. Which is the best way of presenting the results for those who will see them? 
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VIII. Analyze and Interpret Results 

The purpose of most measures is to focus on and uncover problems. 

Examples of key questions to consider are: 

1. Are there undue factors which may have influenced the results (for example, the medication the 
individual was on at the time the measure was administered)? 

2. Is the norm group appropriate and relevant? 

3. Do the results make intuitive sense and fit with your professional judgment/assessment? 

4. Have all the steps been taken to ensure that the most valid inferences can be drawn? 

IX. Select AT(s) and Document Rationale 

The broad purpose of assessment in AT is to assist in the derivation of devices and other 
interventions to enhance a person's functioning and ultimate quality of life. The assessment process 
should reveal obstacles to improved functioning and point the way to areas in need of further 
exploration. It is important to monitor individuals' progress and repeat measures/assessments 
during treatment. When this is done, then it is possible to document outcome attainment. 
Examples of key questions to consider are: 

1 . Has assessment data been balanced with good professional judgment and interpretation and has 
unfounded personal opinion been minimized? 

2. Has the data been used to guide the decision-making of all the appropriate individuals on the 
AT/rehabilitation team, including the consumer? 

In Summary... 

This was just a very brief overview of the considerations involved in the development, 
validation, selection, and use of measures and outcome assessment in AT. Specialized resources 
exist on each of the above topics. Below are some readings relevant to our field that do not 
presume an advanced background in statistics and research methods. 



FOR FURTHER READING 




American Educational Research Association, American Psychological Association, National 
Council on Measurement in Education. (1988). Code of fair testing practices in education. 
Washington, DC: American Psychological Association. 
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American Educational Research Association, American Psychological Association, National 
Council on Measurement in Education. (1985). Standards for educational and psychological 
testing. Washington, DC: American Psychological Association. 

American Psychological Association. (1986). Guidelines for computer-based tests and 
interpretations. Washington, DC: Author. 

American Psychological Association Science Directorate. (1991). Finding information on 
psychological tests. Washington, DC: Author. 

American Psychological Association Science Directorate. (1991). Finding unpublished 
psychological tests and measures. Washington, DC: Author. 

APA Office of E thni c Minority Affairs. (1993). Guidelines for providers of psychological services 
to ethnic, linguistic, and culturally diverse populations. American Psychologist, 48, 45-48. 

Cushman, L.A. & Scherer, M.J. (Eds.). (1995). Psychological Assessment in Medical 

Rehabilitation. Washington, DC: APA Books. 

Dittmar, S.S. & Gresham, G.E. (1997). Functional Assessment and Outcome Measures for the 
Rehabilitation Health Professional. Gaithersburg, MD: Aspen. 

ERIC Clearinghouse on Assessment and Evaluation, The Catholic University of America, 
Department of Education, Washington, DC 20064. 

Johnston, M.V., Keith, R.A., Hinderer, S.R. (1992, December). Measurement standards for 
interdisciplinary medical rehabilitation. Archives of Physical Medicine and Rehabilitation, Special 
Education Issue. 73(12-S). 

Joint Committee on the Standards for Educational and Psychological Testing. (1994, October 7-9). 
Open Conference on the Revision of the Standards: Oral Testimony Summaries; Summary of 
Panel Discussions. Author. 

National Institute on Disability and Rehabilitation Research.(1992). Human measurement in 
rehabilitation. Rehab Brief, 14(2), author. 

In addition to the producers of the above publications, organizations to contact for publications of 
direct relevance to this topic include (a) CARF: The Rehabilitation Accreditation Commission, (b) 
the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) and (c) The National 
Center for Outcomes Research, a program of the Council on Quality and Leadership in Supports for 
People with Disabilities. 
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WHERE ARE WE HEADED WITH ASSISTIVE TECHNOLOGY OUTCOMES? 

Roger O. Smith, Ph.D., OT, FAOTA 
Marcia J. Scherer, Ph.D. 



Dr. Frank DeRuyter noted in his section, "Concepts and Rationale for Accountability in 
Assistive Technology" that, "Accountability today will minimize the denial of goods and services 
tomorrow." It was our guiding premise in assembling these volumes that accountability can be 
achieved by measuring the changes, or outcomes, that result when we provide assistive 
technologies to consumers. Furthermore, we believe that the greatest threat to consumers’ access to 
AT is payors' lack of knowledge about AT outcomes. For years those familiar with assistive 
technology have seen the positive results of assistive technology applications. In fact, the impact of 
assistive technology has seemed so obvious to the field that those closest to assistive technology 
intervention have taken outcome for granted. The coming of the new millennium must change this 

contentedness. 

While the need to measure outcomes of AT is becoming more evident, the means of doing 
so are not always apparent or easy. In fact, both the National Institute on Disability and 
Rehabilitation Research (1995) and the Center for Medical Rehabilitation and Research of the 
National Institutes of Health (1993) have determined that outcomes measurement regarding AT is 
an important research need. Statements from these two agencies clearly call for the 1) enhanced use 
of outcomes measurement systems in practice and 2) the development, testing, and improvement of 
methods for measuring the outcomes of interventions for people with disabilities. 

The application of assistive technology outcomes assessments and outcomes research have 
their own methodological challenges. On one hand, assistive technology outcomes are 
straightforward. Assistive technology, afterall, is tangible and the results seem to be overt. This is 
somewhat deceptive, however. Measuring the impact of assistive technology has its own 
measurement limitations. While some limitations are common to any habilitation or rehabilitation 
intervention, some are specific to assistive technology. 

Issues in the Measurement of A T Outcomes 
The Proof is in the Pudding: Validity 

Successfully documenting outcomes depends on having available appropriate measurement 
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instruments and methodologies. The examples of instruments in Volume 2 can be considered to be 
representative, but certainly not exhaustive. Undoubtedly, there are quality measures being used in 
the field of which we were not made aware. Additionally, new instrumentation and approaches to 
measures will be created (or older measures radically adapted for assistive technology applications). 
But even a cursory review of the measures in Volume 2 reveals that existing instruments tend to be 
very focused in a given practice area of assistive technology or are limited to a particular functional 
domain. In fact, as we delve deeper into the topic of assistive technology outcomes measurement 
we find that it is very complex. While many assistive technology devices show a clear and 
demonstrable impact on the lives of the individuals who use the devices, quantifying this effect 
using a credible process is much more involved than one would think. 

Take for example, construct validity. Construct validity is whether an instrument or an 
approach measures what it hopes to measure. If we measure the outcomes of assistive technology 
use, the question is, are we really measuring the relevant effects of the assistive technology 
intervention? Could the effects result from other related interventions? 

While the overall concept of construct validity is simple, factors related to construct validity 
may be the most challenging aspect of developing sound assistive technology outcomes measures. 
Certainly other c har acteristics of assistive technology outcomes instrumentation are necessary, such 
as content validity and various types of measurement reliability. As the assistive technology field 
begins to tackle outcomes measurement, however, it is the construct issue which has plagued its 
early development. A discussion of the importance of construct validity follows as it affects 
developers when they create new measures and the practitioner when trying to select the most 
appropriate measure to use. 

Fundamental questions must be asked such as," will the outcomes results provide a 
complete and accurate view of reality?" and "what kind of decisions may be made using the results 
of this outcomes information?" Might the results have an impact on the funding of particular types 
of programs or services? For example, if HCFA was to find that a cohort of stroke survivors used a 
walker for an average of a few weeks before improving to the point of discarding the device, they 
might authorize only rental payment and not purchase payment for Medicare recipients requiring 
this type of device. But for those stroke survivors who still require use of a walker and have their 
rental period expire, there may be important negative consequences for their health as well as 
mobility. 

The results from outcomes measurement can also influence decisions made to commence, 
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continue, modify, or discontinue services or programs. For example, if a health maintenance 
organization (HMO) were to find that basic self-care assistive technology devices as applied by one 
service program allowed individuals to return home from a hospital on the average of 1/2 day 
sooner and that they were happier doing this, the HMO could be encouraged to continue referrals to 
this particular rehabilitation program. But that decision would be based on just partial information 
about that program's functioning. Incomplete views of outcomes can also adversely impact future 
services offered to individual consumers as well as future research emphases. 

Selecting the Outcomes to Measure 

DeRuyter suggests that clinical, functional status, quality of life, satisfaction, and cost are all 
important. Smith supplements this categorization, stating that each area of outcome can be elicited 
obtaining objective or subjective data. The subjective data provide the rich perspective from the 
user of the assistive technology. Objective data, on the other hand, reveal how the assistive 
technology performs along more quantitative scales. Both are critical to understand the reality of 
outcome. For these reasons, it is likely that in the near future we will continue to need to cast a 
wide net with our assessments. With this comes the need to determine how many measures we 
need, how focused they need to be and, how fragmented the process becomes as a consequence. It 
appears that we will need many data collection approaches and must then integrate these diverse 
data into the most sensible package. 

Several recommendations are made for today's practitioner about the selection of measures. 

1 . Stay abreast of the increasing number and types of outcome measures. 

2. Consider the breadth and depth of the type of data you need. 

3. Identify instruments which collect a variety of outcome data and 
from more than one perspective. 

4. Plan on collecting data and in a consistent fashion. 

We know two things: a) no outcome are documented without collecting data; b) data are 
worth more when they can be examined across time and users. Consistency is essential. In the 
future, as a field, we hope to identify core data collection instruments and approaches to maximize 
quality and consistency. Hopefully, these will be consolidated into larger databases, which will 
allow scrutiny into informative aspects of cost-benefit. Today, however, we need to begin by 
tending our specific outcome needs. Each individual area of practice needs to begin collecting data 
by finding the most relevant and best instruments available. 
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Teasing Out the Impact of Assistive Technology from the Array of Interventions 

Assistive technology is only one of many powerful approaches to assist people with 
disabilities. In addition to or instead of an AT, environments are accommodated, tasks adapted, or 
assistance is provided by another person (such as a personal assistant, spouse, or parent.) Most 
likely, combinations of all of these interventions are used in combination. Examining the outcomes 
of assistive technology in isolation is only possible by teasing it out from among the combination 
of possible interventions. Yet, there are often interaction effects between these different 
interventions called "parallel intervention" (DePape & Smith, 1987; Angelo & Smith, 1989; Smith 
1991a). The interaction between intervention approaches may be so intertwined that the assistive 
technology cannot be implemented in isolation. If we are looking to design effective outcomes 
measures, we must tease out the specific assistive technology outcomes yet keep it in context with 
parallel interventions. Thus, when we examine the use of outcome instruments, we need to remain 
cognizant of the potential effects of other variables. The use of basic research principles that 
attempt to isolate and control variables will be valuable to substantiate the credibility of our 
outcome data. 

The Variance in Population and Environments 

Assistive technology is applicable to all age groups, individuals with all levels of 
impairment severity, and the full spectrum of etiologies. Moreover, assistive technology is applied 
across a multitude of situations in highly variable environmental contexts. While this seems a 
simple enough statement, the ramifications are somewhat onerous. On one hand, we desire a 
method of assessing assistive technology outcome that is consistent and efficient. Yet, no medical 
diagnosis, functional impairment or disability area, treatment, educational or residential setting, or 
type of assistive technology device is excluded. Almost 20,000 devices and combinations can be 
applied to individuals with all types of impairments, in virtually countless environments and 
situations. As an intervention, assistive technology poses as unique in the broad scope of its 
application. 

Since measures must be specific, yet respect the context in which they are used, two 
implications for practice are clear: a) in all cases when we measure outcomes we need to carefully 
define which population and in which contexts we are measuring. Also, when we scrutinize 
outcomes data and interpret studies, the populations and contexts must be reported in context, b) 
large data-based research studies need to be mounted to examine trends, and identify the true 
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influence of environmental and individual variables. 



The Measurement of Moving Targets 

Outcomes are highly dependent on temporal factors. For example, how well an individual 
might use a manual wheelchair could depend on the activity he had been performing immediately 
prior to the outcomes testing situation. Outdoor weather, how tired the individual is, or natural 
recovery of a medical condition can have a major impact on momentary outcome. While these 
types of variables are easy to understand how they influence outcome, it is often not acknowledged 
how difficult the moving target phenomenon makes true measurement. These transient factors 
often seem so obvious that they are assumed as easy to control as in a formal laboratory research 
study. The problem is that they are much more difficult to control in naturalistic settings. This 
challenges any outcomes measurement system and highlights the importance of multiple data points 

over time. 

Furthermore, the type of assistive technology that might be appropriate is also a moving 
target. One type of technology might be appropriate at a given point in time, but later the assistive 
technology may be obsolete for the individual and their particular needs. This is commonly seen m 
pediatric applications where individuals grow, develop and mature, or with communication devices 
for people with ALS where function slowly degenerates, but there are examples in almost all areas 
of assistive technology application. One might even say that assistive technology should be thought 
of as a temporary intervention. The length of time a given technology is appropriate becomes an 
important factor to measure. 

Towards Our Future and Beyond the Traditional to New Outcomes Measurement Strategies 

Typical outcomes assessments throughout rehabilitation and other fields dealing with 
disability have created assessments where all individuals are evaluated with the same questions and 
comparisons made among persons. Indeed, it is only on this basis that traditional reliability and 
validity assessment has been performed in a normative way. In assistive technology, it has become 
increasingly apparent that due to the variance of people and their situations, standard questions may 
not elicit data important to an individual. Standard questions can skew the results so they do not 
accurately describe the individual's situation. AT outcomes measures of the future must accurately 
target relevant questions for the person and their unique needs. This may mean considering many 
innovative data collection methodologies which do not fit the mainstream of traditional test and 
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measurement theoretical foundations. 

As another example where assistive technology may need to reach beyond traditional 
measurement theory, traditional scales have been dependent upon the objectivity of the data 
collected. The underlying theory is based on the concept that real data can only come from 
something observable. We know, however, that the usefulness of assistive technology is often best 
known by the user. Assistive technology assessment may be prudent to avoid constraining itself to 
objective quantitative methodologies. 

Assessments related to the needs of people with disabilities, and data which are based on 
preferences or opinions, historically have earned little weight. It is difficult to add these numbers 
together or compare the numbers from one individual to another. However, we are seeing some 
innovation in this area. Within the last few years, a number of assessments have been published 
incorporating methods of soliciting subjective data in the form of preferences and opinion. For 
example, in The Matching Person and Technology (MPT) Model (Scherer, 1994) a person is 
compared to him or herself to assess change over time since personal preferences and needs are 
highly individualized. Then AT and other rehabilitation services can be tailored to meet a particular 
person's goals and desires and focus on what needs intervention, not who needs it. The MPT also 
has the capacity to obtain and compare data between users and providers, thus offering a means to 
take into account multiple subjective perspectives. 

Other examples of innovations in AT outcomes measurement are the Trichotomous 
Tailored Sub-branching Scoring (TTSS) system used in OT FACT (Smith, 1992, 1995) and the 
user-centered assessment QUEST (Demers, Lambron, & Ska, 1996) which is based on the 
theoretical framework of the MPT. 

Like AT itself, outcomes research is undergoing innovation. There are many players, yet 
practitioners are key to outcomes research. They are the primary data collectors and may need to 
devote some of their personal time to perform outcomes research if it is not supported by their 
organizations (Jutai, Ladak, Schuller, Naumann, & Wright, 1996). Faculty of professional 
educational programs must teach outcomes research methodology and expect 
assistive technology graduates to understand how to read outcomes research and how to participate 
in the outcomes research process. 

Certification and accreditation agencies must demand an understanding of outcomes 
measurement, theory and application for practitioners working in assistive technology and for the 
programs delivering these services. Researchers and developers performing basic research 
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pertaining to assistive technology must link their studies to functional outcome. Manufacturers and 
suppliers must be cognizant of outcomes studies and ethically promote devices to optimize 
outcomes as opposed to revenues. Assistive technology users must demand outcomes information 
as part of the process for selecting an assistive technology device or an assistive technology service. 
Agencies funding assistive technology services must support the additional expense of collecting 
outcomes data through the design and implementation of outcomes studies. Research funding 
agencies should emphasis the need for the development of innovative outcomes instrumentation 
and the testing and development of large database approaches. 
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ANNOTATED BIBLIOGRAPHY 
on 

ASSISTIVE TECHNOLOGY and OUTCOMES ASSESSMENT 

INTRODUCTION 
Roger Smith, Ph.D., OT, FAOTA 

[Annotated Bibliography follows.) 

Approximately forty papers were identified as key to current discussion of assistive 
technology outcome. While this search was not exhaustive, many of the topics which are relevant 
to assistive technology and outcome were identified, including some found in non-indexed sources 
(fugitive literature). Key articles representing the state-of-the-art in assistive technology outcome 
were annotated in this process. 

The Journal of Assistive Technology was reviewed from 1990 (charter issue) through 1997. 
This uncovered eleven articles pertaining to AT outcomes. Eight articles were found in the journal 
of Technology and Disability from its charter issue through 1997. RESNA annual conference 
proceedings were examined from 1982 through 1997 and located nine papers. Medline, CINAHL, 
Academic Search and ERIC were searched from 1982 to the present revealing nine articles to 
supplement those already found. Team Rehab Report and Rehab Management magazines also 
provided a couple articles. A few other papers were located using references lists within key 
articles as listed above or were identified within the network of practitioners and researchers 
focusing on outcomes in assistive technology. 

This annotated bibliography provides a starting point for a review of outcomes in assistive 
technology. Several themes are easily seen from this brief review of the current literature. 

First, there is a paucity of outcomes research in assistive technology. While research is young in 
many of the rehabilitation-related fields, it is very apparent in assistive technology that outcome 
research and instrumentation is in its infancy. Most authors of the articles reviewed highlight the 
need for more outcomes research, outcomes data collection tools, and lament that more is not 

already done. 

Second, articles related to assistive technology tended to fall into 6 categories: 1) 

Outcomes, 2) Matching the individual and the technology, 3) Quality of Life, 4) Abandonment, 5) 
Device Feature/ Performance, and 6) Cost. From these categories it is clear that the topic with the 
least coverage relates to the relative costs and benefits of assistive technology. 
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ANNOTATED BIBLIOGRAPHY 
on 

ASSISTIVE TECHNOLOGY and OUTCOMES ASSESSMENT 



Prepared by 

Marlene Humbles, BS, OTR 



Albaugh, P.R. & Fayne, H. (1996). The ETPA for predicting technology success with 
learning disabled students: lessons from a multimedia study. Technology and Disability, 

September, 313-318. . 

The study outlined in this paper was designed to test the validity of a modified version of the 
Educational Technology Predisposition Assessment (ETPA) as a tool to measure early adolescents' 
responses to CD-ROM technology with special focus on learning disabled students. The ETPA is a 
self-report form that rates the characteristics of the technology, the individual's temperament, and 
the environment in which the person will use the technology' in four subscales. Results indicated 
that proper assessment of the students would enhance their ability to function in a multimedia 
environment. 

Bain, B., Block, L. & Strehlow, A. (1996). Survey report on the assessment of individuals 
with spinal cord injuries for assistive technology. Technology and Disability, August, 

289-294. ... 

This article points out the current assessment process used by rehabilitation teams working with 
individuals with spinal cord injury in the area of assistive technology. The authors stress that three 
major factors need to be considered to provide individuals with the most appropriate devices: 1) 
the individual's highest level of functioning; 2) the tasks s/he desires to perform; and 3) the required 
level of functioning needed in order to perform those tasks. Various models of assessment were 
introduced that highlighted the different needs of the consumer, i.e., desire to return to work, 
predicted continued use of devices after discharge, more general model for variety of assessment 
needs, the device itself, and the environment in which the device will be used. All of the models 
shared a common emphasis in securing an assistive technology device and include 1) the client 
using the device; 2) the task to be accomplished, 3) the device itself; and 4) the environment. The 
results suggested that in choosing an assistive technology device that clients and care providers 
should be included in every step of the process, clients should become advocates for inclusion in 
the assessment process and that clients should be given the opportunity to try various assistive 
technology devices before a final selection is made. 

Barnicle, K. & Lavanchy, C. (1991). An interdisciplinary approach to product assessment. In 
Presperin, J.J. (Eds). Proceedings of the 14th annual conference. Technology for the 

nineties, 30-32, Washington, D.C.: RESNA Press. 

The importance of having a multi-discipline approach in evaluating the best assistive technology for 
consumers is discussed in this paper. Additionally, the consequences of a good and poor match 
between assistive technology and the user are reviewed. The results emphasized a collaborative 
effort take place between the representatives from disciplines required to evaluate the assistive 
technology and the consumer. 
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Bell, P., & Hinojosa, J. (1995). Perception of the impact of assistive device on daily life of 
three individuals with quadriplegia. Assistive Technology, 7, 87-94. 

A qualitative study addressing the perceptions of three men with spinal cord injuries and their use 
of assistive technology devices in their daily lives was the topic of this article. The results showed 
that the ability to use these devices in their daily activities had a profound impact on their quality of 
life. Emergent themes and "fit" between user and device were discussed. 

Bergen, A.F. (1996). The effect of the mobility assessment process on outcomes: a beginning 
effort. Technology and Disability, May, 17-23. 

The author conducts a study on the effectiveness of outcome studies on services rendered by 
rehabilitation technology suppliers (RTS) of mobility equipment and demonstrate that outcome 
studies are needed to support the role of the RTS. Three questionnaires were mailed out at three 
times during the provision of the mobility equipment and results indicated that greater satisfaction 
resulted in the assessment process when the consumer was involved in the entire process of 
receiving assistive technology products and/or services. 

Brown, D. (1996). Personal implications of functional electrical stimulation standing for 
older adolescents with spinal cord injuries. Technology and Disability, September, 295-311. 

This study investigates the usage of Functional Electrical Stimulation (FES)-augmented in the 
home environment and the effects it may have for increasing the independence of persons with 
Spinal Cord Injury (SCI). One of the basic issues in considering assistive technology devices for 
persons is the overall effect it has on the user. The purpose of this study was to uncover and verify 
the qualities of older adolescent users of FES standing systems and aspects that have the potential to 
add successful use. To investigate these variables, the Offer Self-Image Questionnaire-Revised, die 
Personal Independence Profile, the Craig Handicap Reporting Technique, and the Assistive 
Technology Device Predisposition Assessment tools were used to record the responses of four 
participants. The results showed higher levels of psychological self-sufficiency and physical 
autonomy, cited creative activities for device use reflecting the enabling power of the device in the 
attainment of increased independence, and users felt they could benefit from the use of the device. 
The comparisons are limited due to the small number of participants in study which may not 
generalize the findings to a larger population. 

Cushman, L.A., & Scherer, M.J. (1996). Measuring the relationship of assistive technology 
use, functional status over time, and consumer-therapists perceptions of ATs. Assistive 

Technology, 8, 103-109. „ . . 

This article systematically examined the circumstances surrounding the abandonment of assistive 
technology devices. The authors addressed two consecutive studies and interpreted reasons for 
nonuse of assistive technology devices after a specific follow-up period. Results showed that there 
did not appear to be a straightforward relationship between functional improvement and continued 
use of adaptive devices after discharge, rather, that although functional improvement occurred in 
many of the cases, other reasons for nonuse are still operative. Additionally, the differing views of 
the consumers and the professionals providing the assistive technology services were discussed. 
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Demers, L., Weiss-Lambrou, R., & Ska, B. (1997). Quebec user evaluation of satisfaction 
with assistive technology (QUEST): A new outcome measure. In Sprigle, S. (Ed.) Let's 
tango: Partnering people and technologies. Proceedings of the RESNA '97 annual 

conference, pp.94-96. Arlington: RESNA. 

This paper highlights an assessment tool that evaluators can use to determine satisfaction with 
assistive technology devices and services. The purpose is to provide an outcome measure that will 
reflect a consumer-responsive and client-centered approach that will ensure objective assessment of 
the effectiveness and efficiency of the assistive technology device and/or services. 

DeRuyter, F. (1995). Evaluating outcomes in assistive technology: do we understand the 
commitment? Assistive Technology, 7, 3-16. 

The author points out in this article the critical need for accountability and responsibility as assistive 
technology providers. Rising to the challenge of change to accurately document the outcomes of 
assistive technology devices and services is one of the key issues reflected upon. The author further 
identifies the categories of what to evaluate and the factors influencing on how assistive technology 
devices and services will be measured. 

DeRuyter, F. (1994). Assistive technology usage outcome: A preliminary report. In Binion, 
M. (Eds). Proceedings of the RESNA '94 annual conference. Tuning in to the 21st century 
through assistive technology. Listen to the music, 27, Arlington, VA: RESNA Press. 

This article addresses a project that was developed to begin to examine the underlying elements of 
assistive technology devices and/or services that may contribute to the outcomes. The author 
stresses the importance of accountability at levels in which assistive technology is delivered. 

Dowler, D.L., Hirsh, A.E., Kittle, R.D., & Hendricks, D.J. (1996). Outcomes of reasonable 
accommodations in the workplace. Technology and Disability, 5, 345-354. 

A study was conducted to determine if reasonable accommodations in the workplace were made in 
accordance with ADA and if they were perceived as being effective. A variety of disabling 
conditions was included in this study. The results indicated that the cost figures for these 
accommodations as reported by employers were consistent with accommodation cost information 
found in the literature. 

Fuhrer, M. (1997). Evaluating the outcomes of assistive technologies and related services: 
the roles of program management-oriented and rehabilitation science-oriented outcomes 
research. 

Paper presented at the ann ual AAATE meeting in Thessalonaki. The author outlines two different 
approaches to outcomes research in assistive technology: program management-oriented and 

rehabilitation-oriented approaches. The two approaches are compared in terms of their aims, 
audiences' choice of outcome criteria, characteristic strategies, financial support, and information 
dissemination practices. 

Gitlin, L. (1996). Factors predicting assistive device use in the home by older people 
following rehabilitation. Journal of Aging and Health, November, 8 (4), 554-576. 

The goal of this study was to examine the use of assistive technology devices for one particular 
group of elderly that were discharged from rehabilitation to their homes. Factors influencing 
device use in the home, characteristics of consistent device users and potential reasons for 
abandonment were discussed. 
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Gitlin, L. (1995). Why older people accept or reject assistive technology. Generations, 19(1), 
41 - 46 . 

The author discusses in this paper the use and abandon use of assistive technology devices and 
services. Factors that influence assistive technology use/nonuse such as person-based, nature of the 
environment and characteristics surrounding the device itself were reviewed. Factors that predict 
use, the user’s perspective, and reasoning for abandoning the devices were highlighted. 

Gitlin, L.N., Levine, R., & Geiger, C., (1993). Adaptive device use by older adults with mixed 
disabilities. Archives of Physical Medicine and Rehabilitation, 74, (2), 149-152. 

Two descriptive pilot studies were conducted to further examine the use of assistive devices by 
older adults were addressed in paper. The first pilot study focused on device use by older adults 
discharged from rehabilitation. The findings from this first study indicated that strategies for 
performing self-care activities may differ in the home from those that were performed in the 
hospital. The second pilot study focused on home care therapists working with die elderly with 
disability. The findings from this study indicated that relatively little or no coordination between 
inpatient rehabilitation training approaches and home care training in adaptive devices existed. 
Three interrelated factors were discussed as to why decrease in device use occurred in these two 
studies. 

Gitlin, L., & Schemm, R. (1996). Maximizing assistive device use among older adults. Team 

Rehab Report, 7, (4), 25-28. . 

With the growing number of elderly patients, the need for assistive technology devices and services 
will increase. Reasons why older adults abandon assistive devices were discussed and potential 
solutions to decrease the abandonment rate given. Guidelines to help increase the rate of device use 
were reviewed with a list of helpful questions to ask. 

Gordon, W. A., Brown, M.M., & Raquarsson, K. (1982). Assessing the impact of 
technological innovations on the daily life of persons with disabilities. In Bowman, B.R. 
(Eds). Proceedings on the fifth annual conference on rehabilitation engineering, 8, Bethesda, 

MD: Rehabilitation Engineering Society of North America. 

This article addresses the concern of the impact of technological innovations on the user. This 
study described the impact that environmental control units and their use or nonuse had on the daily 
activities performed by 20 high-level quadriplegics. Results showed that consumers who were 
provided sufficient information about the use of the units were more likely to purchase and continue 
to use the unit to increase their optimum functioning. 

Hammel, J. (1996). What's the outcome? Multiple variables complicate the measurement of 
assistive technology outcomes. Rehab Management: The Interdisciplinary Journal of 

Rehabilitation, February/March, 97-99. 

This article addresses the need to find ways of measuring the effectiveness and efficiency of 
assistive technology systems. "An outcome can be defined as a natural result or consequence and 
should be measured over time". Consumers and payors both are interested in cost-effectiveness 
data and the impact of assistive technology and services have on the quality of life. Earlier studies 
have focused solely on the impairment or pathology level of impact of assistive technology, 
whereas the impact of assistive technology on the functional task performance and the cost 
trade-offs affiliated with the improved performance is just beginning to be studied. Issues that 
include the accounting for multiple uses of assistive technology, long-term impact, abandonment 
issues, and which instruments should be used when measuring assistive technology outcomes are 
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further addressed. 



Heerkens, W.D., Briggs, J., & Weider, T.G. (1997). Using peer mentors to facilitate the 
match of person and technology. In Lawrence, D.L. (Eds). Let’s tango: Partnering people 
and technologies. Proceedings of the RESNA ’97 annual conference, pp. 484-486. Arlington 
VA: RESNA Press. 

This article addresses the successful outcomes achieved by persons who have been mentored prior 
to selecting an assistive technology device and / or service. It details how the mentors effectively 
employ the Matching Person and Technology assessment along with other assessments that will 
empower individuals to make the best choice for their assistive technology needs. 



Hoyer, E.A., & Brandt, W.D. (1983). Devices to aid in independent living. In Bowman, 

B.R., McNeal, D.R., & Monzingo, D.W. (Eds). Proceedings of the sixth annual conference on 
rehabilitation engineering, pp. 19-21. Bethesda, MD: Rehabilitation Engineering Society of 
North America. 

The authors address the need to evaluate communication devices and ensure successful use of 
devices by including the user in the evaluative process. The ease of which the features could be 
used, the cost and service of the assistive technology devices are key in identifying compliance by 
the consumer. 

Jutai, J., Ladak, N., Schuller, R., Naumann, S., & Wright, M. (1996). Outcomes 
measurement of assistive technologies: An institutional case study. Assistive Technoloev, 8, 
110-120. 

This paper examines the assistive technology use in the context of total rehabilitation and the 
institutional culture in which services are delivered. Examples of outcome priorities have been 
identified along with the tools and approaches used in the MacMillan site of the Bloorview 
MacMillan Centre. The outcomes associated with assistive technology included the purpose of the 
outcomes measurement, the definition of the outcomes and the ease of implementation. Case 
studies were presented. 

Keefe, B., Scherer, M.J., & McKee, B.G. ( 1996). Mainepoint: Outcomes of teaching 
American sign language via distance learning. Technology and Disability, September, 
319-326. 

This study highlights the delivery of instruction through the use of multimedia technology of 
computers, televisions, VCRs, and cordless phones. Students participated in a research study to 
complete coursework in American Sign Language via distance learning. The characteristics of 
students who successfully completed the course were compared to students who did not 
successfully complete the course. The Tennessee Self Concept Scale, Learning Styles Inventory, 
Survey of Technology Use and Educational Technology Predisposition Assessment were utilized to 
determine the characteristics. The Survey of Technology Use and the Educational Predisposition 
Assessment had consistent results regarding the profile of the successful students and can therefore 
potentially replace the more expensive tests of the Tennessee Self Concept Scale and the Learning 
Styles Inventory. 
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Kohn, J. G., LeBlanc, M., Mortola, P. (1994). Measuring quality and performance of 
assistive technology: Results of a prospective monitoring program. Assistive Technology, 6, 
120-125. 

In order to determine the satisfaction with an assistive technology device, a follow-up program is 
required to accurately reflect the outcome (i.e., satisfaction, device performance) the device may 
have on the individual. This paper investigated the effects a follow-up program would have on a 
population of 60 consumers from Lucille Salter Packard Children's Hospital at Stanford as well on 
a replication study. This program was instituted as a quality assurance tool in order to make 
changes in the assistive technology device services as needed. Results showed that the follow-up 
program was successful in documenting device user feedback and the outcome measures were 
helpful in determining satisfaction and device performance. 

Mann, W.C., Hurren, D., Tomita, M., & Charvat, B. (1996). Use of assistive devices for 
bathing by elderly who are not institutionalized. The Occupational Therapy Journal of 
Research, 16, (4), 261-286. 

The authors developed a hierarchical model for use in prescribing bathing devices related to 
mobility and support. The problems experienced with their use, satisfaction or dissatisfaction with 
the bath device, and matching the best bath device to the user were noted. The overall importance 
of careful measurement and assessment of needs were emphasized. 



McGrath, P.J., Goodman, J. T., Cunningham, J., MacDonald, B.J., Nichols, T.A., & Unruh, 
A. (1985). Assistive devices: Utilization by children. Archives of Physical Medicine and 
Rehabilitation, 66, (7), 430-432. 

The use of assistive technology devices by children was the subject of this study. It addresses the 
manner in which the devices were prescribed-from a variety of professionals and the purchase of 
the items from a variety of commercial vendors. Reasons for utilization were reviewed. The results 
revealed that a high percentage of devices prescribed were used a large percentage of the time and 
that satisfaction rate was high. 



Merbitz, C. (1996). Frequency measures of behavior for assistive technology and 
rehabilitation. Assistive Technology, 8, 121-130. 

This paper reviews a new approach that uses natural science measures and a model (selectionism) 
from the field of Behavior Analysis to document the outcomes of assistive technology. This 
method of selectionism has been used successfully in the field of education and the benefits are 
noted in empirically defined measurable outcomes. It is also useful for measuring change during 
treatment. The author emphasizes further pilot studies to determine the efficacy and effectiveness 
of this method when determining the outcome of assistive technology and to further the success of 
implementation of these ideas. 



Minkel, J. (1996). Assistive technology and outcome measurement: Where do we begin? 
Technology and Disability, July, 285-288. 

This article emphasizes the importance of what a single service provider can do to measure the 
actual results of assistive technology devices and services they are delivering. Dr. Frank DeRuyter 
(1995) describes five dimensions of outcome measurement that can be used with varying 
audiences: 1) clinical results, 2) functional status, 3) quality of life, 4) satisfaction, and 5) cost. 
The researcher can choose a dimension that is most important and relevant and measure the 
changes in that dimension after identifying the audience the outcome measure is targeted. A 
comparison of assistive technology devices to the measurement of changes in clinical results is 
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reviewed along with an analysis of different methods to achieve an accurate picture of consumer 
satisfaction with assistive technology is also discussed. The author points out that more than one 
dimension of satisfaction needs to be measured, such as changes in functional status and analysis of 
costs as well as consumer satisfaction as noted in completed surveys. The outcome measure is 
dependent on the vantage point of consumer, provider and/or payor and it is necessary to take these 
roles into consideration when measuring outcomes. 

Nordenskiold, U., Grimby, G., Hedberg, M., Wright, B., & Linacre, J. (1996). The structure 
of an instrument for assessing the effects of assistive devices and altered working methods in 
women with rheumatoid arthritis. Arthritis Care and Research, 9(5), 358-367. 

The authors used a newly developed instrument, the Evaluation of Daily Activities Questionnaire 
(EDAQ) in order to analyze the effects of using assistive devices or altered working methods on a 
group of twenty-one women with rheumatoid arthritis. The results showed that the EDAQ can be a 
useful instrument in assessing the effectiveness that assistive devices have on the impact of daily 
activities. * 

Oldridge, N. B. (1996). Outcomes measurement: Health-related quality of life. Assistive 
Technology, 8, 82-93. 

This study addresses the need for the evaluation of use of assistive technology devices to the 
health-related quality of life for people with disabilities. There is increasing importance to include 
the patient perception of usefulness and satisfaction of devices as related to their quality of life. The 
proponents of assistive technology claim that these technologies "improve patient quality of life", 
however, have difficulty in providing objective, measurable methods to document these effects. The 
author points out that generic and specific health-related quality of life instruments exist and could 
be potentially useful when examining outcome measures of assistive technology devices. 

Oldridge, N.B. (1996). Outcomes measurement: Health state preferences and economic 
evaluation. Assistive Technology, 8, 94-102. 

Accountability of effectiveness of assistive technology devices is addressed in this article. With the 
increasing concern for economic evaluation of health care technologies and the impact they have 
on the quality of life years gained, it is important to develop techniques that include the cost and 
consequences of assistive technology devices. The author points out that an economic evaluation 
will help assistive technologists in making the most informed decisions in use of alternative 
assistive technology options. Further, the examination of the relationships between the cost of their 
interventions and the effectiveness of their interventions need to be addressed and documented. 

Parette, H.P. (1996). Augmentative and alternative communication impact on families: 
trends and future directions. Journal of Special Education, 30(1), 77-99. 

The issues surrounding the family concerns prior to the prescription of augmentative and 
alternative communication (AAC) for children with disabilities is reviewed in this paper. AAC 
abandonment pattern characterizations were discussed along with implementation of AAC across 
multiple contexts, maintenance of AAC equipment and future directions. 

Phillips, B., & Zhao, H. (1993). Predictors of assistive technology abandonment. Assistive 
Technology, 5, 36-45. 

Reasons for technology abandonment were discussed in this article. The purpose of this study was 
to determine the reasons why the assistive technology users accept or reject the devices. The 
research focused on three areas of device abandonment that involve the users' personal 
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characteristics and technology acceptance, device attributes consumers prefer, and device utilization 
surveys. The findings of this study suggested that services that involved the consumer input and 
accommodating their needs on a long-term basis would enhance consumer satisfaction and reduce 
abandonment of the device. 

Pritham, C.H. ( 1996). Guest editorial: Quality improvement process/outcomes research. 
Journal of Rehabilitation Research and Development, 33(3), vii-viii. 

The author discusses the quality improvement process currently existing in the area of outcomes in 
the healthcare field. The emphasis is to include the patient and his/her family members first to 
determine the quality of services rendered, how services are rendered within the system and how 
the patient will eventually return to the community and state of wellness. Focusing on qualitative 
research to encompass the complex nature on how the person perceives the outcomes from 
healthcare services is stressed. 

Russel, M. (1993). Continuous quality improvement in assistive technology- Establishing, 
maintaining, measuring. In Binion, M. (Eds). Proceedings of the RESNA '93 annual 
conference. Engineering the ADA: from vision to reality with technology, 14-16, Washington, 
D.C. : RESNA Press. 

This paper addresses the Continuous Quality Improvement (CQI) strategies that have taken root in 
rehabilitation programs and emphasizes he shift that assistive technology services must focus on in 
order to provide quality services/products. Measuring the outcomes of services/products issued, 
customer focus, and defining the service strategy are key issues discussed. 

Scherer, M.J. ( 1996). Outcomes of assistive technology use on quality of life. Disability and 
Rehabilitation, 18(9), 439-448. 

A review of the use and abandonment of assistive technologies along with results of research 
studies on the use of assistive devices are discussed in this article. The author articulates the 
influences on technology abandonment and stresses the importance of considering the consumer's 
perspective when selecting an assistive device to enhance their quality of life. Further, additional 
areas that require more study are detailed with a special emphasis for researchers to encompass a 
more comprehensive view of assistive technology. 

Scherer, M.J., & McKee, B.G. (1994). Assessing predispositions to technology use in special 
education: music education majors score with the "Survey on Technology Use". In Binion, 
M. (Eds). Proceedings of the RESNA '94 annual conference. Tuning in to the 21st century 
through assistive technology. Listen to the music, 194-196, Arlington, VA: RESNA Press. 

The puipose of this article was to show that by matching the person with the correct technology 
using the Survey on Technology Use for students in music education, an accurate predisposition to 
technology is assessed. Students were able to identify an increased number of educational 
opportunities available to them as a result of this assessment. 

Smith, R. (1996). Measuring the outcomes of assistive technology: Challenge and innovation. 
Assistive Technology, 8(2), 71-81. 

Successfully documenting the outcomes of assistive technology use is presented in this paper. 
Methods to collect outcome data and discussion of new measurement strategies for assistive 
technology along with research designs were reviewed. Contributions of implementation of 
outcome strategies were emphasized. 
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Smith, R.O. (1993). Assessing the impact of assistive technology using OT FACT Version 2.0. 
In Binion, M. (Eds). Proceedings of the RESNA '93 annual conference. Engineering the 
ADA: from vision to reality with technology, 186-188, Washington, D.C.: RESNA Press. 

The author discusses an outcome instrument that would be useful in measuring the impact of 
assistive technology. OT FACT is described in detail and the potential for the theoretical elements 
of this perspective to be compatible with a discipline-free assistive technology philosophy is 
reviewed. 

Smith, R.O. (1996). Measuring the outcomes of assistive technology: Challenge and 
innovation. Assistive Technology, (8), 71-81. 

This article describes the need for outcome measurement tools in assistive technology interventions 
and highlights the issues relevant to meet these needs. The author addresses five categories that 
appear to be common to discussions of outcomes data dimensions that include 1) Performance of 
interventions; 2) User performance; 3) Support environment resources and performance; 4) Cost; 
and 5) Combinations of outcomes data. The author also discusses the challenges that rise when 
trying to isolate the particular impact of assistive technology devices from all other possible 
interventions; the variance that exists in the populations and environments; the wide variance in 
assistive technology services; measurement of dynamic variables; the dependence of assistive 
technology on other areas of the profession; and the number of studies on usage of assistive 
technology devices that focus solely on the issue of compliance. The importance of outcomes 
research methodology along with outcomes measurement is stressed. 

Van der Loos, H.F. & Hammel, J. (1996). Engineering reasonable accommodation: the 
delivery and use of assistive technology in a vocational training program. Technology and 
Disability, 5, 371-382. 

This paper addressed the use of assistive technology that would assist the needs and capabilities of 
the individual student with learning disabilities using a three- workstation classroom designed by the 
Vocational Training Facility (VTF). The study focused on the integration of the assistive 
technology into the workstation and the reasonable accommodations solutions offered to empower 
the consumer. The results showed that if a reasonable accommodation was done well and focused 
on empowering the consumer to help them make the best choice, which a successful impact on the 
entire process of integrating the assistive technology to the consumer was made. 
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Annotated Bibliography by Category 



The following table sorts the annotated bibliography in the following categories: outcomes, matching person with 
technology, quality of life, abandonment, device feature/performance and cost. 



OUTCOMES 

Definition: The quality of both assistive 
technology services and devices is 
determined by measurement of outcomes 
resulting from both the provision of the 
services and devices and the utilization of 
the technologies to facilitate functional 
improvement and quality of life for the 
individual consumer. Outcome measures 
evaluate the end result of the assistive 
technology intervention (Cook & Hussey, 
1995). The following articles addressed 
outcome measures issues: 

Bergen, A.F.(1996) 

Demers, L., Weiss-Lambrou, R., & 

Ska, B. (1997) 

DeRuyter, F. (1995) 

Fuhrer, M. (1997) 

Hammel, J. (1996) 

Jutai, J., Ladak, N., Schuller, R., 
Naumann, S., & Wright, M. (1996) 
Keefe, B., Scherer, M J., & McKee, 

B.G. (1996). 

Kohn, J.G., LeBlanc, M., & Mortola, 

P. (1994) 

Merbitz, C. (1996) 

Minkel, J. (1996) 

Oldridge, N.B. (1996) 

Oldridge, N.B. (1996) 

Pritham, C.H. (1996) 

Smith, R. (1996) 

Smith, R.O. (1993) 

Smith, R.O. (1996 


MATCHING PERSON WITH 
TECHNOLOGY 

Definition: Involves the appropriate 
matching of person to technology. The 
three components of matching the person to 
the technology include the person who will 
be the user of the technology, the 
technology itself and the milieu or 
environments) in which the user interacts 
with the technology (Scherer, 1996). The 
following articles addressed matching 
person with technology issues: 

Albaugh, P.R. & Fayne, H. (1996) 

Bain, B., Block, L., & Strehlow, H. 

(1996) 

Bamicle, K. & Lavanchy, C. (1991) 
Heerkens, W.D., Briggs, J., & Weider, 

T.G. (1997) 

Mann, W.C., Hurren, D. Tomita, M. & 
Charvat, B. (1996) 

McGrath, P.J., Goodman, J.T., 

Cunningham, J., MacDonald, B.J., 
Nichols, T.A., & Unruh, A. (1985) 
Scherer, M.J., & McKee, B.G. (1994) 


OUALITY OF LIFE 
Definition: Quality of life has been 
described as life satisfaction, well-being 
and general affect. It has been studied by 
looking at subjective reports along with 
such objective indicators as 
sociodemographic and medical factors, 
functional impairment, and ability to 
work.( Scherer, 1996). 

The following articles addressed quality of 
life issues: 

Bell,P. & Hinojosa, J. (1995) 

Brown, D. (1996) 

Scherer, M.J. (1996) 


ABANDONMENT 

Definition: Device abandonment research 
falls into three important areas of study: 
users’ personal characteristics and 
technology acceptance, device attributes 
consumers prefer, and device utilization 
surveys. (Phillips, & Zhao, 1993) The 
following articles addressed abandonment 
issues: 

Cushman, L.A. & Scherer, M.J. (1996) 
Gitlin, L. (1996) 

Gitlin, L. (1995) 

Gitlin, L.N., Levine, R., & Geiger, C. 
(1993) 

Gordon, W.A. , Brown, M.M., & 
Raquarsson, K. (1982) 

Parette, H.P. (1996) 

Phillips, B., & Zhao, H. (1993) 

Scherer, M.J. (1996) 


DEVICE FEATURE/ 

PERFORMANCE 

Definition: These are standards that 
specify how a device should perform and a 
set of tests to be used as a basis for 
comparing similar products from different 
manufacturers (Cook & Hussey, 1995). 
The following articles addressed device 
feature/performance issues: 

Hoyer, EA., & Brandt, W.D. (1983) 
Nordenskiold, U., Grimby, G., 

Hedberg, M., Wright, B., & Linacre, 

J. (1996) 

Russel, M. (1993) 

Van der Loos, H.F. & Hammel, J. 

(1996) 
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COST 

Definition: Involves labor cost and 
production costs. The following article 
addressed cost issue: 

Dowler, D.L., Hirsh, A.E., Kittle, 

R.D., & Hendricks, D.J. (1996) 
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Citations for Definitions found in Table of Annotated Bibliography by Category 

Quality of Life Category - from Scherer, M.J. (1996). Outcomes of assistive technology use on 
quality of life. Disability and Rehabilitation, 1 8(9), page 441 . 

Matching Person with Technology Category - from Scherer, M.J. (1996). Outcomes of assistive 
technology use on quality of life. Disability and Rehabilitation, 18(9). This definition was 
paraphrased from her definitions on pages 445-446. 



Cost Category - from Dowler, D.L., Hirsh, A.E., Kittle, R.D., & Hendricks, D.J. (1996). 
Outcomes for reasonable accommodations in the workplace. Technology and Disability, 5. This 
definition was paraphrased from the outcome - cost of accommodation on page 350. 

Outcomes Category, Device Feature/Performance Category, and Abandonment Category from the 
Phillips and Zhao 1993 article. 
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